Final Evaluation Strengthen integrated early warning systems for more effective disaster risk reduction in the Caribbean through knowledge and tool transfer

Report Cover Image
Evaluation Plan:
2018-2021, RBLAC
Evaluation Type:
Final Project
Planned End Date:
02/2019
Completion Date:
04/2019
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
15,000

Share

Document Type Language Size Status Downloads
Download document TORs -Project Evaluation Consultancy, final.pdf tor English 502.83 KB Posted 325
Download document Evaluation Report_EWS_Caribbean_DIPECHO_final.pdf report English 9559.65 KB Posted 530
Title Final Evaluation Strengthen integrated early warning systems for more effective disaster risk reduction in the Caribbean through knowledge and tool transfer
Atlas Project Number:
Evaluation Plan: 2018-2021, RBLAC
Evaluation Type: Final Project
Status: Completed
Completion Date: 04/2019
Planned End Date: 02/2019
Management Response: Yes
Focus Area:
  • 1. Poverty
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 3.3.1 Evidence-based assessment and planning tools and mechanisms applied to enable implementation of gender-sensitive and risk-informed prevention and preparedness to limit the impact of natural hazards and pandemics and promote peaceful, just and inclusive societies
Evaluation Budget(US $): 15,000
Source of Funding: EU-ECHO
Evaluation Expenditure(US $): 16,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Nana Gibradze Team Leader GEORGIA, REPUBLIC OF
GEF Evaluation: No
Key Stakeholders: ECHO, IRFC, CDEMA, OXFAM
Lessons
1.

EWS fall within the bigger picture of preparing communities, and should not be considered as a standalone exercise, but need to be integrated into wider Risk Reduction programmes and targeted interventions based on the needs of the target communities and capacities of institutions. As such, the tools transfer was relevant as it provided a set of customizable tools that can be integrated into comprehensive national and regional risk reduction programmes in the Caribbean. In this regard, it is of utmost importance to further highlight the social aspects of the risk reduction, taking into account different types of vulnerabilities and social profiles of target communities.


2.

Cultural and political differences between Cuba and the beneficiary countries need to be taken into account when designing experience transfers. Cuban training methodology, while yielding results, was not necessarily the most optimal and user-friendly in that it relied mostly on the unilateral delivery of technical information and little practical work and interaction. While this may have been partly due to the time constraints, understanding cultural environments of the beneficiary institutions is essential for maximizing the benefits of the knowledge transfer and eventual sustainability of the results.

It is important to have adequate knowledge of the capacities of the national institutions, especially of those related to response and recovery and of the effects of disasters on their functioning and project implementation. Knowledge of the capacities of national institutions is also of utmost importance for the experience transfer. The success of know-how transfer/SSC does not depend solely on the offering country but the capacity of the recipient to provide quality information related to adequate preparation, carry out the planned activities and take ownership of the experience. It is therefore important to provide training on the South-South Cooperation processes and formats not only to the offering institution but the recipients as well to ensure successful adoption of the knowledge and practices.


3.

Sharing practices and tools effectively implies condensing years of accumulated experience and knowledge into a limited project timeframe and training format. Installing new structures and strengthening capacities in institutions is an incremental process and may take years. It is therefore important to define the scope and time of the knowledge transfer more realistically, factoring in the challenges and delays related to institutional characteristics, different procurement and implementation timeframes related to travel and workshop preparation, human resource capacities and the like, to avoid possible incompletion and sustainability risks.


Tag: Operational Efficiency

4.

Gender mainstreaming remains a challenging and sensitive concept not only among the institutions but some implementing agencies as well. It is therefore important to simultaneously address the issue of gender equality and linkages between gender and disaster risk not only with the beneficiaries but with partners as well, to ensure proper integration of gender-sensitive contents in project activities and foster a stronger commitment to and follow-up on gender mainstreaming agenda. Identification and strengthening of gender focal points in partner and beneficiary institutions is key for stronger gender equality action.


Findings
1.

6.1. Relevance

Based on the data obtained from the desk review, interviews and surveys, the project was considered as timely and relevant, both at national and regional levels, as it responded to the priority needs embedded in the national and regional policies and programmes and complemented the advances of the previous efforts to strengthen the EWS in the region. According to one respondent, its“particular relevance is in going beyond the disaster risk and fostering a better understanding of the multiple hazards and their linkages with climate change and its effects on community resilience”.

The project was viewed as timely as it consolidated years of technical assistance provided by ECHO, UNDP and other agencies for the strengthening of EWS and focused on institutionalizing the capacities and knowledge in recipient countries. Its timeliness was particularly valued as it allowed to assess the challenges and develop the solutions in the context of the hurricanes Irma and Maria that occurred during the project implementation and generated better understanding.

 


Tag: Disaster risk management Relevance National Regional

2.

One aspect of the project’s relevance for participating countries is in its focus on strengthening national and local capacities and awareness through the transfer of knowledge and know-how and sharing of experiences. The project was viewed as valuable for closing knowledge gaps and identifying areas for future interventions for national disaster management agencies. The Road maps developed in the framework of the project, were perceived as particularly relevant as they are customized based on the gaps analysis and prioritized with the active participation of national entities, rather than in response to donor priorities. In that regard, all beneficiary countries considered national ownership of the process as another important characteristic of the project, distinguishing it from other similar interventions. In the words of one respondent from Antigua and Barbuda, “the [EWS-related projects] have been around for a long time and [national authorities] will insist in not accepting projects for projects’sake, they should fit into country priority, otherwise, it is not value-added and takes too much effort”.


Tag: Knowledge management Ownership Capacity Building

3.

The participating stakeholders considered the checklist as one of the most relevant and useful tools provided by the project. The respondents particularly appreciated the gaps analysis as it allowed establishing a certain baseline for future work and identify new challenges and areas of work beyond the existing knowledge. According to the stakeholders, the gap analysis allowed the national institutions to identify bottlenecks and develop specific interventions, improve inter-institutional integration and information sharing. The gaps analysis also takes stock of the existing capacities and resources and allows for better assessment of institutional needs. As noted by a respondent from Dominica, “the gaps analysis revealed two things: people of different sectors were not aware of the existing assets and achievements; the disaster office had more knowledge because of working with UNDP’.


Tag: Relevance Awareness raising National Institutions

4.

The three-tier approach (regional, national and community level actions) and engagement of different institutions for each tier were considered as appropriateby stakeholders. Most of the respondents valued the community engagement and considered volunteer observer model of the Cuban experience as an important asset. However, at least one stakeholder in Saint Vincent and the Grenadines was not in favor of the volunteer approach noting that while volunteering is a strong tradition in Cuba linked to its particular governance and societal organization, volunteer culture is lacking in Saint Vincent and the Grenadines can not be relied upon for early warning. Instead, the respondent advocated for automatic real-time stations for monitoring river levels to minimize errors and related risks and improve data quality.

At the national level, the project was valued for strengthening Pillar One of EWS by developing hazard, vulnerability and risk maps and Pillar Two of EWS by strengthening capacities and tools for developing own forecasting models, more precise and continuous monitoring and generate tailor-made products for the general public and relevant institutions. Respondents valued the project for increasing the awareness of hazards and finding solutions not previously tried before, which may not eliminate the risk completely but increase the resilience by knowing the risks and finding ways of adapting.


Tag: Knowledge management Project and Programme management South-South Cooperation Disaster Risk Reduction Relevance National Regional

5.

While the overall project and most of the interventions, processes and tools were considered timely, adequate and needed, some respondents questioned the extent to which national authorities of participating countries consider them relevant, referring to the signing of the agreements during the High-level Handover event in St. Luciaby high-ranking decision-makers representatives of only countries: St. Lucia (host), Cuba and the Dominican Republic. This low turnout of decision-makers was considered by some of the interviewed respondents as an indicator of a relative lack of commitment to the project and its results; others considered it not indicative, one respondent noting that “it is rather normal for ministers not to participate in events outside their countries” due to human resource constraints and other commitments. See 6.4.for further details on the Handover Event.

Triangulation of the data does not confirm the stakeholder perception of low attendance: given the 50% turnout, the attendance was satisfactory, especially considering that the Handover meeting was attended by representatives of the participating country, including Haiti, as well as Oxfam and CREWS. This is particularly noteworthy given (i) sustainability and linkages with CREWS actions in the region, commitment to EWS and the uptake of Roadmaps as guides; (ii) the pan-Caribbeanrelevance of the EWS tools highlighted by the participation of the Haitian Civil Defense; (iii)importance of sharing OXFAM’s results and handover with the broader regional audience.


Tag: Challenges Relevance Human and Financial resources National Institutions

6.

6.2. Effectiveness

6.2.1. Achievement of planned results

Evidence obtained from the reports available by the time of the evaluation, and stakeholder responses, indicates that the project has completed all of the planned activities achieving most output-level results, and, according to the interviewed stakeholders, had a positive effect on the participating countries. The review of output targets and indicators suggests 100 % achievement of the results and 100%execution of the budget.

6.2.1.1. Result 1

UNDP, in close coordination with CDEMA and IFRC led the improvement of the EWS Toolkit (Caribbean Regional EWS Toolkit)with the 1) addition of the Spanish version of existing case studies, 2) revision of the webpage based on feedback from stakeholders and project partners, 3) significant changes to the layout (including the mobile interface compatibility), and 4) inclusion of additional tools from CDEMA and Cuba such as: 1) Standard for conducting hazard mapping, vulnerability assessment and economic valuation for risk assessment for the tourism sector, 2) Guide for the Development of National Disaster Risk Management Strategies for the Tourism Sector in the Caribbean, 3) Guide for the Development of National Disaster Risk Management Strategies for the TourismSector in the Caribbean, 4) Disaster Risk Management Strategies and Plan of Action for the Tourism Sector in the Caribbean, 5) Manual for Baseline Data Collection at the Country Level, 6) Model National Hazard Mitigation Policy for the Caribbean, Preparation of a National Hazard Mitigation Policy: Guidance Document, 7) Model Comprehensive Disaster Management Legislation and Regulations 2013 and Adaptation Guide, 8) Disaster Information Kit for the Media, 9) Family Disaster Plan, 10) Model National Operations Readiness Checklist, 11) Model National Evacuation Plan, and 12) CDEMA Model National Evacuation Plan, 13) CDEMA Model Safe School Programme Toolkit User Guide, 14) Model Integrated Relief Policy and 15) Model Integrated Relief Plan along with Cuban tools used during the implementation of priority actions.


Tag: Disaster risk management Effectiveness Communication Knowledge management Coordination

7.

6.2.1.2. Result 2 and 3

The project supported five countries13 in carrying out national assessments to identify the gaps in their Early Warning Systems. These assessments were guided by the Multi-Hazard Early Warning System Checklist, an 88-question tool developed through a consultation process among the International Network for Multi-Hazard Early Warning Systems (IN-MHEWS) partners during the ‘Multi-hazard Early Warning Conference’ in Mexico in 2017, and then adapted to the Caribbean in 2018. The Checklist, which is structured around the four key EWS elements, contains the main components and actions to which national governments can refer to when developing or evaluating early warning systems. Each target country and their National Disaster Management Systems assessed their EWS based on the checklist.

National Validation Workshops were carried out in all the countries, where key national actors reviewed the information compiled in the Check lists forming the basis for the national EWS Gap Reports.The validation process helped confirm the quality and validity of the collected data, build a consensus on the identified gaps, and identify further steps. The findings of the assessments were transformed into priority actions to address specific gaps and to guide national and local efforts and investments, towards improving and strengthening integrated multi-hazard early warning systems, through the formulation of a national EWS Roadmap.


Tag: Disaster risk management Effectiveness Knowledge management Vulnerable

8.

The project carried out sustainability dialogues in order to raise awareness on the importance of MHEWS to validate the roadmaps developed by the project. These dialogues, organized by CDEMA in St. Vincent and the Grenadines on 16 October, 2018, Antigua and Barbuda on 26 October, 2018, Saint Lucia on 6 December, 2018 and Dominica on 9 January, 2019, had active participation of relevant national institutions and laid bases for future actions towards enhanced EWS and increased resilience to disaster risks.

The project carried out the systematization: Strengthening Early Warning Systems in the Caribbean, which examines two key aspects of the project: (i) results and lessons of the national assessment and planning processes to strengthen Early Warning Systems in each country, fostered by the project; and (ii)results, lessons, processes and tools used to implement selected Roadmap priority actions through South-South Cooperation (SSC) between the target Caribbean countries and Cuba. The systematization covers the EWS process, including the checklist, gap validation and roadmap; South-South Cooperation process; and 5 case studies of priority actions carried out in Antigua and Barbuda, Dominica, Dominican Republic16, St. Lucia, and St. Vincent and the Grenadines.


Tag: Effectiveness Awareness raising South-South Cooperation National Institutions

9.

6.2.1.3. Result 4

Further to the technical assistance provided to countries by IFRC, CDEMA and Cuban experts, described under the Result 3, the project carried out communication, visibility and advocacy actions to promote the integrated MHEWS approach and uptake of EWS toolkits and experiences.

Through its advocacy actions, the project promoted the integrated MHEWS approach and the importance of enhancing the MHEWS at national and community levels at various regional forums, reaching approximately 262,312 persons. These actions include:

  • The CDM Conference in coordination with CDEMA and OXFAM, that included an EWS parallel session on EWS lessons learned after Hurricane season 2017 in the Caribbean;
  • Regional Seminar on DRR exchanges in Dominica Republic, which included the presentation of Cuban EWS toolkit by Cuban representatives and presentation of the CREWS initiative on lessons learned from past 2017 Hurricane Season in the Caribbean by WMO;
  • LAC Region DIPECHO partners workshop,which included the presentation of “adaptation, institutionalization and application of MH EWS checklist”, and presentation on strengthening knowledge and working techniques on EWS through SSC promotion;
  • DRR Regional Platform for the Americas, where UNDP and CDEMA promoted project achievements on good practices for early warning systems for different hazards and priority investment opportunities to strengthen Early Warning Systems in Small Island Developing States;
  • CDEMA TAC meeting, where MH EWS checklist was presented and endorsed by the TAC team.
  • Civil Defense International Congress on Disasters in Cuba, where UNDP promoted project achievements and presented the Cuban EWS toolkit; and, solution packages designed by Cuban expert committee to four Caribbean recipient countries. The director of DRM system from SaintLucia and Dominican Republic received the documents during the session.

Tag: Effectiveness Communication Knowledge management

10.

The project developed an external communication strategy that was aimed at four different types of audience: (i) Caribbean DRM structures, governmental DRM entities from seven countries, organizations relevant in the humanitarian and development decision making processes, EWS entities, Red Cross National Societies, Municipal and community PMR committees, selected communities; (ii) general population from seven countries involved in the project as well as those from the rest of the Caribbean region; (iii) donor community interested in supporting disaster risk reduction & recovery; and (iv) private sector companies/Non-Governmental Organizations that will be looking to work on DRR and recovery.

The strategy had two main objectives: (i) Advocacy, that emphasized a partnership-oriented approach (Sendai/CDM strategy/EWS consortium) for a comprehensive achievement of DRR results across the board and motivating authorities to invest, work and take over the responsibility of owning an effective integrated EWS at regional and national level; and, (ii) Communications, to raise awareness on EWS tools to increase the access to them at regional, national and community levels, sensitize different target groups regarding the need of strengthening the EWS aspects as one of the most effective life-saving tool, and increase the knowledge and awareness of EWS tool/actions by the authorities and population including its 4 pillars.


Tag: Effectiveness Communication Partnership Advocacy

11.

The majority of the stakeholders were of the opinion that the project has not only achieved its results but given the challenges and constraints has exceeded expectations. In the words of one respondent, “if we look at the level of complexity the project has overachieved the results and even though a lot has to be done in the second phase the achievements of the project have been tremendous”.

The overwhelming majority of the stakeholders value highly the quality of the tools and knowledge transferred by the project and consider them as valuable assets for strengthening national EWS capacities for effective disaster risk reduction. The stakeholders credit the transfer process for equipping the national institutions and the implementing partners with important tools, capacities and knowledge to further expand the outreach and strengthen national and regional capacities; integrating the community vulnerability and capacity assessment processes and response plans; opening communication channels between the countries and identifying pathways for potential bilateral partnerships.


Tag: Challenges Effectiveness National Regional South-South Cooperation

12.

Several respondents noted that while the results have been achieved successfully, there was no time for translating them into action and “operationalizing” them through more specific decision-making. Despite the successful adaptation and transfer of the tools and their high potential impact on reducing disaster risk, it was not possible to test their actual performance through application. As reported by some stakeholders, some of the countries have not yet been able to fully test the transferred tools and capacities; therefore, in light of the absence of data, it was considered too early for the evaluation to assess their effectiveness and potential impact.

Survey results indicate that 17 of the 21 respondents (81%) consider that the project has achieved its results. 2 do not know, 3 respondents -partially or to a marginal extent, and 1 responded –I would like to know the indicators used for measuring the results. As regards the question if the project has had a positive effect onits beneficiaries, 20 out of 23 (95%) beneficiaries responded Yes; and 1 –Don’t know, 1 –Still unable to fully determine and 1 –did not respond.


Tag: Effectiveness Communication Data and Statistics

13.

6.2. Effectiveness

6.2.2. Coordination

The project had a Project Board composed of the Director of the UNDP Regional Hub for LAC, representing the UNDP Regional Bureau for LAC; representatives of national disaster management offices of Dominican Republic and Saint Vincent and the Grenadines, representing the beneficiary countries; ECHO coordinator for the Caribbean, representing ECHO; UNDP Sustainable Development and Resilience Team Leader, Regional Advisor on Disaster Risk Reduction, Regional Project Coordinator, UNDP Barbados Deputy Resident Representative and/or Climate Change and Disaster Risk Reduction Project Officer, UNDP Dominican Republic Deputy Resident Representative and/or Disaster Risk Reduction Project Officer, UNDP Cuba Deputy Resident Representative and/or Disaster Risk Reduction Project Officer, IFRC Deputy Regional Director, Head of Caribbean Country Cluster, and/or Project Officer and CDEMA Executive Director, Deputy Director, and/or project officer, representing the implementing partners. The Board was in charge of overall policy and technical guidance, inputs to and approval of plans, budgets and schedules, changes, requests and monitoring, decision-making, arbitration, oversight and endorsements. The Project Board met once in April 2018and is set to meet at the end of June2019to formally close the current phase and review the progress of the next phase.


Tag: Oversight Partnership Project and Programme management

14.

Roles of implementing agencies were partly informed by their institutional profiles and scope of action, with each agency playing a leading role in one or more beneficiary countries17 and supporting actions at regional, national, and community levels18. Thus, as part of its regional coordination role, CDEMA led the adaptation of the EWS Checklist and supporting guidance documents such as concept documents, agendas and reporting formats to support its application; sustainability dialogues in four beneficiary CDEMA Participating States (PS); and the Multi-hazard Early Warning System (MHEWS): Achievements and Strategic Path Forward High-level Hand-over Event.

In addition to leading the implementation of the Project in Saint Lucia, CDEMA supported the improvement of the EWS Toolkit with the expertise of the Webmaster who also supports the updating on statistics on the toolkit; administration of the surveys on awareness of EWS tools; and the planning and coordination meetings for the efficient implementation of the Project. CDEMA led coordination efforts with UNDP and IFRC to consolidate key methodological material and project-related information on its website, which includes the improved MHEWS in three languages (English, French and Spanish)19. The MHEWS Checklist is available in digital format to Project beneficiaries and particularly the partners who attended the High-level handover meeting held in Saint Lucia in February 2019.


Tag: Communication Project and Programme management Coordination

15.

Most of the interviewed stakeholders who were aware of the project coordination mechanisms considered it mostly adequate, efficient and meriting replication in the second phase, despite certain deficiencies. In the words of one respondent, “monthly coordination kept us on our toes, it was a good mechanism to keep implementation on target and while it was annoyingly intense in the beginning, stepping back it was a good decision to have these different layers of coordination that ensured simultaneity and coherence of project interventions”.

According to the interviewed stakeholders, the interagency collaboration was the biggest success from the organizational perspective, some calling it “phenomenal”and “never seen in other organizations”. According to the respondents, project coordinators worked well as a team, were open to new ideas, were solutions-oriented and spoke with common messages with the countries, maintaining a unified message and approach with different stakeholders. As noted by one respondent, “it was a kind of a team that makes a lot of difference especially when working with multiple partners, levels and countries”.


Tag: Coordination

16.

6.2.3. Synergies with other initiatives

The project established internal and external synergies with various initiatives and projects implemented in the region.

The collaboration agreement between UNDP, CDEMA and IFRC, allowed enhancing regional linkages and dialogue about DRR and EWS approaches: Partnership with CDEMA supported long-term regional leadership in EWS, commitment of governments to EWS and sustainability of the investment and mainstreaming the CDM Strategy in the project. Partnership with IFRC ensured a comprehensive approach on strengthening EWS and a greater iteration of EWS at national and local level.

Within the project framework, IFRC established internal synergies with its other programmes and funding initiatives.i) CEWS interventions in the IFRC’s Community Resilience Building Programme via support of Global Affairs Canada and the Canadian Red Cross, and the OFDA. ii) Provision and testing of CAP software for Antigua and Barbuda through the IFRC Hurricane Irma Appeal. ii) partnership between the Met Service and Dominica Red with CIMH to identify, procure, install and test hazard monitoring instruments for floods and landslides.


Tag: National Regional Communication Knowledge management Partnership Programme Synergy Coordination

17.

In collaboration with Cuba’s EWS Expert Committee, HVR studies and lessons learned from similar experiences in Dominica and Saint Lucia were translated into French to contribute to the external SSC process between Cuba and Haiti.Also, in the Dominican Republic, UNDP coordinated with the Cuba EWS committee and WFPto strengthen weather forecasting capacities through the training on the use of the numerical model WRF/ARW.

In Dominican Republic, the project supported the organization of the Regional summit for Exchanging innovate experiences and new practices on disaster risk with the EU-funded project“Strengthening the organizational and functional structures of Disaster Risk Management in the Dominican Republic”, coordinated by the National Emergency Commission (CNE) of Dominican Republic, the Directorate-General for Multilateral Cooperation (DIGECOOM).

 


Tag: Effectiveness Communication Knowledge management Capacity Building

18.

6.2.4. Gender and vulnerability

The project had made special provisions for gender mainstreaming in all project activities and ensured that gender and other vulnerability factors and gender-sensitive language were included in the checklist and community assessment criteria, needs of women and vulnerable groups were taken into account during the revision and selection of tools proposed to enhance the integrated EWS, as well as in the improvement of the online and community-based Early Warning toolkit and training. The project was rated as GEN2.

To ensure proper gender mainstreaming in all project activities, the project sought the involvement of the Regional Gender Team of UNDP Regional Hub early on. The Gender Team reported participation in the pre-PAC and PAC meetings and revising the project document, suggesting ways to incorporate mainstreaming analysis. Suggestions for gender-sensitive activities and indicators were shared with CDEMA and IFRC. A concept note was developed that outlined specific actions and products for mainstreaming gender in project activities through the strategic and technical support of the Gender Team, review of the products, activities and events to guarantee uniformity and coherence and effective incorporation of the gender perspective.


Tag: Gender Mainstreaming Knowledge management Vulnerable Effectiveness

19.

As reported to the evaluation, as a result of this technical support, the roadmaps for Saint Vincent and the Grenadines and the Dominican Republic achieved a higher degree of gender mainstreaming, whereas Antigua and Barbuda, Dominica and Saint Lucia, relatively little. It is understood that this was related to the quality of gender analysis carried out prior to the development of roadmaps in each country. Overall, most respondents point to increased consideration of gender issues in the discussions, tools and processes supported by the project and appreciate the inclusion of gender-sensitive criteria in the checklist and solution packages.

Despite the achievements, the evaluation encountered some weaknesses/issues related to gender and inclusion as reported by stakeholders:

While each agency had appointed a gender focal point to review event agendas and key documentation and ensure gender mainstreaming, level of commitment and quality of work of these focal points varied and required assistance from the regional Gender Team;

  • The Gender Team was not invited to project meetings as their participation was not envisaged. Neither did Gender Team collaborate with the technical team from Cuba, since it was understood that the original Cuban toolkit model could not be altered and gender mainstreaming could be addressed through the checklist and the roadmaps;
  • The inclusion in some countries of key actors like representatives and entities for people with disabilities and senior citizens was considered of critical importance; however, these actors were not evenly represented across all countries, whereas groups working gender equality and women empowerment were largely absent despite the specific emphasis of the Checklist on this issue;
  • As reported by stakeholders, the process of gaps analysis and roadmap elaboration in various countries revealed that gender and vulnerabilities related to age and disability, social and economic inequalities are not always considered in the local EWS and preparedness and response actions, as reported by some stakeholders. Gender and vulnerability gaps are common in the knowledge of the DRR, hazard and risk maps; however, data is not always disaggregated by gender, age and other vulnerability criteria, are obsolete or not quantified.

Tag: Challenges Effectiveness Gender Mainstreaming Technical Support Vulnerable

20.

6.3. Efficiency and implementation challenges, weaknesses and strengths

The analysis of available information indicates that the project funds were reasonable for the implementation of planned actions. While additional funds would have certainly allowed for expanding some country-specific actions and procuring additional goods and services (e.g. software for NODS in Antigua and Barbuda or water level monitoring equipment in Saint Vincent and the Grenadines23), the evaluation did not find any evidence of a significant shortage of funds affecting the achievement of the planned results.

In terms of financial efficiency, the obtained evidence suggests that the project had executed 98% of programmed funds. While the project did not envisage any resource mobilization efforts, in Antigua and Barbuda, project funds were complemented by funding obtained by the Red Cross through the emergency appeal in the aftermath of the hurricanes and funded part of the required software.

The total cost of the project personnel was 511,201 Euros, which constitutes 37% of the total project cost.


Tag: Efficiency Resource mobilization

21.

In terms of availability of human resources, the project implementation was rather stretched, as most of the national beneficiary institutions, as well as some implementing agencies have relatively small teams and the same personnel has to address various issues. Thus, during the hurricanes, most of the counterparts were engaged in emergency response and recovery actions and had to temporarily suspend project-related activities. Even in the absence of hurricanes, national disaster management institutions’ human resource capacities are rather limited and to a certain extent, affect project implementations. Same persons in national disaster management agencies often act as focal points for different areas. Staff turnover was also an obstacle noted by various stakeholders. In Dominica, project focal points changed threetimes from project inception without adequate handover and notice.

Some respondents reported human resource shortages well, putting a strain on the project team and complicating coordination due to the staff work overload. Evidence suggests that these shortages were not relatedtolimited funding given the availableallocation for human resources, but to the inefficientplanning of human resources.


Tag: Disaster risk management Effectiveness Human and Financial resources

22.

Time was also scarce given the delays that affected the project implementation. As reported by several stakeholders, the project activities was suspended for three months in the first week of September due to the hurricanes. The project planning meeting of December 2017 adjusted the workplans and the project effectively commenced in January 201 8accomplishing most of the planned actions in 15 months as evidenced by Project documentation and acknowledged by stakeholders. Although the majority of the respondents consider that without hurricanes, the project would have completed the planned activities, most respondents, including the Cuban experts, agree that the project timetable and scope were rather ambitious, especially with regards to the time allocated for experience transfer in the recipient countries.

According to the Cuban respondents, the available time and financing were not sufficient for properly accomplishing the task given the magnitude of the proposed actions: technical experts considered that there was not enough time for trainings and for carrying out vulnerability studies, which were completed in less than two weeks in the beneficiary countries, whereas it takes up to two years to accomplish the same task in Cuba. The Cuban experts pointed out that two weeks is not enough to fully transfer the know-how for carrying out the vulnerability studies and the countries will need continued support. Also, many beneficiaries and the Cuban experts consider that the time allocated for training in Cuba was not sufficient to achieve the expected results and added value.


Tag: Challenges Programme/Project Design

23.

This section describes the challenges reported by the respondents and identified in the documents that were beyond the control of the project management and were related to environmental factors, donor requirements, institutional capacities and availability of information and the like.

The most significant challenge was the occurrence of the Hurricanes Irma and Maria, which affected the participating countries and diverted the national human resources to response and recovery efforts. National disaster agencies were fully vested in the process but were distracted by the hurricanes and functioning in recovery mode for most of the project’s initial phase. Even the countries not directly affected by hurricanes participated in recovery efforts which led to slowing down the implementation and caused a significant delay in the project implementation and low execution rates in the first year of the project.

Some respondents reported a relatively low level of participation of national institutions as was the case in Dominica, where the nature of key stakeholder involvement was described more as a coordination of the consultants’ work, rather than deep involvement and participation. The pace of national implementation and human resource limitations were viewed as a permanent challenge, often due to staff shortages and/or absence of technical personnel that did not allow proper participation in the project and at times required the hiring of national consultants.


Tag: Challenges Human and Financial resources National Institutions

24.

6.3.2.1. Exogenous challenges

The project design was also a challenging process due to changes in the initial proposal to ECHO and implementation setup, related to the late inclusion of IFRC at the suggestion of ECHO. The feedback and the request to include IFRC coincided with the Montreal Disaster Risk Reduction Platform Conference where UNDP staff were expected to participate. Likewise, the initial UNDP proposal included Saint Vincent and the Grenadines, Dominican Republic, Dominica and Cuba whereas ECHO requested the inclusion of Haiti and suggested that some of the islands prioritized by IFRC (in their original proposal) be included. The final proposal to ECHO was reformulated in a week; this request for quick turnaround does not allow for proper planning, coordination and participation of all stakeholders and adequate reworking of the budget given the time limitations.

The project encountered challenges related to the quality of the information provided for the development of priority actions. As noted by some respondents, several countries did not provide sufficient and quality information; GIS data for hazard, risk and vulnerability studies in Saint Lucia and Saint Vincent and Dominica were not accessible. In Saint Vincent and the Grenadines, the final training was postponed till March 2019 given that the data provided to Cuban experts was not accurate and had to be collected again25. Because of the delays and quality of data, some field missions, which were supposed to provide data for solution packages, were also delayed; as aresult, some initial solution packages were developed before the field missions and initially lacked coherence, the latter partly due to the fact that in the absence of quality data and limited access to internet, Cuban experts had to develop the solutions on the basis of their limited knowledge of the context.


Tag: Challenges Project and Programme management

25.

Respondents report challenges related to coordination of events, such as sustainability dialogues between the implementing agencies and national institutions that affected preparation and participation. This was mainly due to the complexity of the project and difficulties in coordinating agendas among stakeholders in the different countries. In general, the implementation pace differed between countries, depending on the size and human resource capacities of participating institutions. To the extent possible, the project mitigated these deficiencies by providing national consultants in support of national disaster management institutions.

The overwhelming majority of respondents in all countries except the Dominican Republic considered the language barrier as a significant impediment, especially during the trainings, when project managers had to step in to translate. In addition to the language-related obstacles, the respondents mention several aspects of the SSC as challenging: (i) the methodology for experience transfer and pedagogical methods applied by the Cuban experts during trainings did not properly take into account the cultural specificities of the recipient countries and were considered as lacking dynamic interaction and feedback; (ii) Development of solutions packages was based on the Cuban model and were at times difficult to adapt to different political and social environments of the recipient countries; (iii) The composition of the Cuban team would change at different stages of transfer, which affected proper understanding of the issue and continuity.


Tag: Challenges Promotion of dialogue Coordination

26.

This section describes challenges inherent to the implementing agencies and partners, their management structure, rules and regulations that govern the implementation of the project.

Despite the overall success of the multi-agency approach of the project, harmonizing the mechanisms and processes between agencies was challenging. The three implementing agencies have different implementation and reporting rules 26, and administrative procedures, which complicated monitoring and affected coordination as financial and other information was not always available simultaneously. As reported by some beneficiaries, despite numerous monitoring trips and project-related travel carried out by the project coordination unit, the unit staff were not always able to process information and respond to assistance requests from national authorities, especially in the aftermath of the hurricanes. In order to streamline implementation, inter-agency agreements had to be adjusted.

Another important reported challenge was the distribution of funds implemented by agencies. Project funds were not distributed to agencies by results but based on the countries they covered and implied mixed management of results, whereby several agencies would be responsible for specific output results. This arrangement was directly related to ECHO’s request to incorporate IFRC, which did not allow sufficient time to rearrange the entire project structure and organize the results in a way that would be more appropriate to each agencies’ niche areas. Therefore, all partners were incorporated into the existing project design and financial architecture, which complicated budgeting and planning, especially given that UNDP had minimal time to develop a new proposal after the inclusion of IFRC. This rushed proposal development limited the possibility to properly discuss and design the budget and complicated project management and reporting.


Tag: Challenges Efficiency Operational Efficiency Partnership Policies & Procedures

27.

6.3.2.3 Strengths

Data obtained from the interviews, the survey and some project material allow to identify a series of strengths that contributed to the achievement of the results and could be translated into replicable good practices.

One of the most often mentioned strengths is the sharing practices/horizontal transfer/SSC process. Most respondents agree that despite certain obstacles and delays, the horizontal transfer was a success, that allowed the participating institutions to engage in experience sharing and learning from Cuban experience and developing potentially long-lasting partnerships. The respondents considered every component of the SSC process as the major strength of the project, highlighting the importance of this experience for the success of the project. Support provided by UNDP-hired SSC consultant was considered a highly positive factor in the successful implementation of the SSC process.

Linked to the SSC is the country-driven character of the project, noting the emphasis on strong institutional participation in the identification of the gaps and design of solutions and engagement of diverse actors in the process. Respondents also note the three-tier approach as highly positive and innovative, fostering collaboration between agencies and the communities, building rapport and enabling better institutional integration at different levels. The respondents considered the division of responsibilities between the three implementing agencies as well-thought and useful, taking advantage of each agency’s comparative advantages and strengths.


Tag: Coordination South-South Cooperation

28.

6.3.2.4. Weaknesses

As mentioned above, the language barrier was considered a major weakness related to the horizontal/SSC; while fully aware of the impossibility of avoiding challenges caused by the bilingual nature of the project, the respondents called for better handling of translation and interpretation needs, providing well-translated documents and considering the associated delays and costs in the planning.

Despite highly valuing the established coordination mechanisms and processes, the respondents highlighted certain weaknesses in coordination and exchange of information between institutions, especially related to events and travel; as noted by several respondents, planning of events was challenging since stakeholders’ agendas were not always available.

The inclusion in some countries of key actors like representatives and entities for people with disabilities and senior citizens was considered of critical importance, however, it could have been promoted more evenly across all countries as well as other key missing actors like groups related to gender equality or women empowerment who were largely absent despite the emphasis of the Checklist on this issue specifically. Another key element that most felt needed to be reinforced was the local-level participation, given that some of the key gaps are related to the vertical integration between national and local levels.


Tag: Challenges National Regional National Institutions

29.

6.4. Sustainability

This chapter provides the evaluation’s findings on the potential sustainability of the project results and challenges thereof, based on the respondents’ opinions and the revised material. This chapter does not offer the evaluator’s appraisal of the sustainability of the results, which will be offered in Chapter 7. Conclusions.

The project had a strong sustainability aspect incorporated in its logicas it was considered the last DIPECHO-funded EWS project in the Caribbean and to a certain extent, was meant to provide exit strategies after years of implementation. The intervention logic built on the strengthening the four pillars of EWS and through this integrated approach offered foundations for sustainability. Trainings, experience transfer and toolkits were aimed at strengthening institutional capacities and awareness while the gap analysis and development of solution packages intended to entrench the acquired know-how and instruments in national institutions.


Tag: Sustainability Ownership Awareness raising

30.

Another important element of project sustainability was the sustainability dialogue process carried out in all participating countries. A key commitment made during the dialogues was the need to present the roadmaps to permanent secretaries in order to advocate for improved legal and institutional arrangements, the human and financial support required and other considerations to allow for integration and improvement of the MHEWS. Other commitments included the inclusion of the EWS Roadmap actions into the Cabinet approved three-year Work Program of the National Office of Disaster Service (NODS) 2019-2021 and sustained conversation on EWS among stakeholders in Antigua and Barbuda. The NODSis already making use of the CAP to the benefit of communities and has demonstrated a commitment to EWS in their 3-year work program. The NODS also has the support of the Line Minister for sustaining the EWS.

In Saint Vincent and the Grenadines, a review of the national roadmap is proposed as an initial task of the EWS Committee and implemented as appropriate. In Saint Lucia, the commitment was made to integrate priority actions into the work programme and programmes of partner agencies, advance the legal and institutional arrangements for MHEWS and the integration of 51work with the disabled. For Dominica, key areas to consider for advancing the MHEWS relate to the institutional and legislative framework, sectoral and programming considerations.


Tag: Sustainability Human and Financial resources Promotion of dialogue

31.

During the high-level handover of the project to beneficiary institutions, the attending DRM directors reiterated their commitment to sustain the actions and strengthen of MH-EWS. High-level decision-makers from Cuba, Dominican Republic and St Luciasignedthe Statements of Commitment, including the Statement of Commitment“Sustaining the Multi-Hazard Early Warning System Strengthening Process in Saint Lucia”, signed by the Prime Minister. These signatures were mentioned as indications of ownership and willingness to sustaining and expanding the achieved results.

Due to their absence at the Handover meeting, the directors of NODS of Antigua and Barbuda, NEMO of Saint Vincent and the Grenadines and ODM Dominica couldn ´t sign the commitment agreements.By the time of the writing of this report, stakeholders from the remaining countries had been discussing the date for signing additional commitments.It is understood that the Hon. Minister for Social Transformation and Human Resource Development of Antigua and Barbuda29has committed to the advancement of EWS by the NODS through its workplan for the next 3years. Based on the commitments manifested during the sustainability dialogues and some interviews, it is expected that the remaining countries will also commit to sustaining and implementing the project results. CDEMA has agreed to follow up on the signatures and it is expected to reach the lacking 3 commitments in the first half of 2019.


Tag: Challenges Sustainability Human and Financial resources Technical Support

32.

Institutionalization of tools and processes was also considered challenging, mainly due to the standard 18-month duration of DIPECHO projects, which may be sufficient for the implementation of actions but not sufficient for providing proper advocacy and support to institutions for their appropriation by the countries. The online toolkit updated in the framework of the project is considered as a base product which can be further adapted and upgraded with new elements as relevant for each Disaster Managementagency but will require stronger institutional support and ownership by the national bodies and CDEMAin order to be sustained.

Some stakeholders identified potential obstacles to sustainability-related to inadequate legal and institutional frameworks on which the MHEWS depend. For example, in Dominica, there is no legal framework to implement many recommendations; the country will have to set up institutional frameworks and structures to ensure the follow-up. Some countries also indicated the need to put in place legislation to engage private sector in the implementation of the EWS.


Tag: Challenges Sustainability Rule of law

33.

According to some respondents, this is related to the relatively less importance given to disaster management issues in some countries of English-speaking Caribbean. As mentioned by one respondent, “the problem is that in a region with constant disaster risks, there are no funds for disaster management as the issue is still considered of secondary importance in some countries”.The respondents note, that lack of funding affects the monitoring and data collection, upkeep of infrastructure and upgrades of technologies that are essential for sustaining the results and ensuring the proper functioning of the EWS in the countries.

Several stakeholders, including national actors and implementing agencies/partners, considered the absence of the high-level representatives at the Handover event as an indication of the lack of political commitment in these countries. However, evidence of sufficient participation of the leading institutions and government agencies in national sustainability dialogues suggests a certain level of ownership and political will in all beneficiary countries, which needs further reinforcement through institutional mechanisms for proper sustainability.


Tag: Disaster risk management Challenges Sustainability Ownership Data and Statistics

Recommendations
1

Ensure the value, timeliness and relevancy of tools and know-how offered by the project and south-south cooperation mechanisms employed for knowledge generation and sharing.

2

Establish effective coordination, communication and horizontal transfer mechanisms to achieve a satisfactory level.

3

Ensure efficiency of time and human resources to avoid challenges and delays.

4

Plan for exogenous challenges that can manifest in the course of the project’s implementation as much as possible.

5

Plan for sustainability despite challenges related to funding, institutional capacities, legal backing and political will.

 

Management Response Documents
1. Recommendation:

Ensure the value, timeliness and relevancy of tools and know-how offered by the project and south-south cooperation mechanisms employed for knowledge generation and sharing.

Management Response: [Added: 2020/04/28] [Last Updated: 2020/12/26]

See key actions.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Eligible countries for SSC identified their own needs and gaps and identified how SSC could be helpful to them
[Added: 2020/04/28]
National counter parts / Project coordinator 2019/09 Completed Saint Lucia, Antigua and Barbuda
1.2 The tools developed by the Cuban team are easily contextualized to the reality of other SIDS
[Added: 2020/04/28]
UNDP Cuba/ Project Coordinator 2020/04 Completed Note for time frame: Their relevance will continue beyond project implementation phase History
2. Recommendation:

Establish effective coordination, communication and horizontal transfer mechanisms to achieve a satisfactory level.

Management Response: [Added: 2020/04/28] [Last Updated: 2020/12/26]

See key actions.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Adoption of TEAMS folder sharing platform for partners to have access to all relevant project documentation and to upload requested information.
[Added: 2020/04/28]
Partners / Project coordinator 2019/12 Completed
2.2. Monthly technical meetings with partners to ensure good coordination amongst them
[Added: 2020/04/28]
Project coordinator 2020/04 Completed Note for time frame: Entire duration of project implementation phase
2.3 Project Board Meeting that gathers all partners and their national counterparts
[Added: 2020/04/28]
Project coordinator 2018/12 Completed
3. Recommendation:

Ensure efficiency of time and human resources to avoid challenges and delays.

Management Response: [Added: 2020/04/28] [Last Updated: 2020/12/26]

See key actions.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Monthly technical meetings with partners to ensure good coordination amongst them
[Added: 2020/04/28]
Project coordinator 2020/04 Completed Note time frame: Entire duration of project implementation phase History
3.2 Regular reminders to partners to take into account the timetable for internal administrative processes
[Added: 2020/04/28]
Project coordinator 2020/04 Completed Note for Time frame: Entire duration of project implementation phase Comments: Through emails, Skype calls and whatsapp messages. History
4. Recommendation:

Plan for exogenous challenges that can manifest in the course of the project’s implementation as much as possible.

Management Response: [Added: 2020/04/28] [Last Updated: 2020/12/26]

See key actions.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1Analyze the calendar of national and community events to identify potential delays (elections, carnival, end-of year holidays)
[Added: 2020/04/28]
Project coordinator 2020/04 Completed Note for time frame: Entire duration of project implementation phase History
4.2 Regular monitoring of hydro meteorological events in the region
[Added: 2020/04/28]
Partners/ Project coordinator 2020/04 Completed Note for time frame: Entire duration of project implementation phase History
5. Recommendation:

Plan for sustainability despite challenges related to funding, institutional capacities, legal backing and political will.

 

Management Response: [Added: 2020/04/28] [Last Updated: 2020/12/26]

See key actions.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Ensure project ownership from national counterparts
[Added: 2020/04/28]
Partners / Project coordinator 2020/04 Completed Note time frame: Entire duration of project implementation phase History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org