Terminal Evaluation of the Strengthening Climate Information and Early Warning Systems project

Report Cover Image
Evaluation Plan:
2012-2018, Malawi
Evaluation Type:
Project
Planned End Date:
07/2018
Completion Date:
12/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
35,000

Share

Document Type Language Size Status Downloads
Download document Final TE ToR of International Consultant (PIMS 5092)dated 1st Feb 2018.docx tor English 74.53 KB Posted 497
Download document Final TE ToR of local consultant (PIMS 5092).docx tor English 76.18 KB Posted 531
Download document Early Warning System Project Terminal Evaluation Report.docx report English 465.64 KB Posted 679
Title Terminal Evaluation of the Strengthening Climate Information and Early Warning Systems project
Atlas Project Number: 00077203
Evaluation Plan: 2012-2018, Malawi
Evaluation Type: Project
Status: Completed
Completion Date: 12/2018
Planned End Date: 07/2018
Management Response: Yes
Focus Area:
  • 1. Environment & Sustainable Development
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 5.4. Preparedness systems in place to effectively address the consequences of and response to natural hazards (e.g. geo-physical and climate related) and man-made crisis at all levels of government and community
SDG Goal
  • Goal 13. Take urgent action to combat climate change and its impacts
SDG Target
  • 13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries
  • 13.2 Integrate climate change measures into national policies, strategies and planning
  • 13.3 Improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning
Evaluation Budget(US $): 35,000
Source of Funding: GEF
Evaluation Expenditure(US $): 25,130
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Jiri Zeman Mr. jirkazeman@seznam.cz CZECH REPUBLIC
Wlton Phalira Mr. w.phalira@yahoo.com
GEF Evaluation: Yes
GEF Project Title: Terminal Evaluation of the Strengthening Climate Information and Early Warning Systems project
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: FSP
GEF Phase: GEF-5
GEF Project ID: 4994
PIMS Number: 5092
Key Stakeholders: Department of Disaster Management Affairs (DoDMA); Department of Climate Change and Meteorological Services and GEF
Countries: MALAWI
Lessons
Findings
1.

Project relevance

The EWS Project was designed in accordance with relevant national policies and international commitments of Malawi, as well as with donors’ policies and assistance programs, including:

  • 2006 Malawi National Adaptation Programmes of Action (NAPA) and its priorities:
    • Project priority 3:        Improving agricultural production under erratic rains and changing climatic conditions (improved early warning systems)
    • Project priority 4:        Improving Malawi’s preparedness to cope with droughts and floods (strengthening drought and flood forecasting and early warning systems through human and technical capacity building)
    • Project priority 5:        Improving climate monitoring to enhance Malawi’s early warning capability and decision making and sustainable utilization of Lake Malawi and lakeshore areas resources (climate monitoring and early warning systems for Lake Malawi and lakeshore areas for improving pre-disaster preparedness of rural fishing and farming communities)

Tag: Agriculture Climate Change Adaptation Disaster risk management Relevance Capacity Building

2.

1.1.1Project implementation approach

The EWS Project was designed with a strategy to strengthen the

  • Generation and use of reliable weather/climate information and
  • Early warning systems in Malawi

through improving national capacities to generate and use weather/climate information in planning for, and management of:

  • climate hazards and
  • long-term strategic planning.

 


Tag: Agriculture Climate Change Adaptation Disaster risk management Natural Disaster Relevance Capacity Building

3.

1.1.1Log-frame analysis

The LogFrame matrix follows in principle the project implementation strategy designed in the Project Document.

The project LogFrame includes in total 6 indicators and 9 end-of-project targets with 20 individual specific EOP targets for project objective and outcomes, as well as baseline, source of verification, and risks and assumptions.

The LogFrame does not specify midterm targets. This undermined monitoring of progress during project implementation period.

The project objective indicator “Capacity as per capacity assessment scorecard” includes 36 additional capacity indicators of departments to produce, package, and disseminate weather/EWS information, and of legislative and governance framework, and for each indicator a baseline and target values are established. There is no information provided on methodology to be used for assessment, nor methodology used during the baseline assessment. Thus, it is not possible to replicate the same methodology at the end-of-project assessment in order to receive comparable results. This type of assessment is also very subjective: it depends on individuals providing the assessment. Thus, the capacity assessment scorecard has very low informational value. It is hardly measurable in an objective way, and thus it is not suitable to serve as a LogFrame SMART indicator to measure cumulative project achievements.


Tag: Disaster risk management Natural Disaster Relevance Monitoring and Evaluation Results-Based Management

4.

Outcome 2 indicator “Percentage of population with access to improved climate information and flood, drought and Mwera wind warnings (disaggregated by gender)”, and its baseline and target, are not SMART/specific enough. Weather forecasts and warnings have been published by DCCMS/DWR/DoDMA via media, including radio broadcasters in Malawi, even before the EWS Project, although the quality and frequency of forecasts was significantly lower. All communities and most families have had access to radio broadcasting, including weather forecasts in English and Chichewa, well above the target value of 17% of population.


Tag: Climate Change Adaptation Natural Disaster Impact

5.

1.1.1Assumptions and risks

The LogFrame included assumptions and risks for the project objective and for both project outcomes. In addition to the six risks identified and described in the Risk Log, the LogFrame specifies one additional risk.

The Risk Log in the Annex 8 of the Project Document specified six main risks, including their potential consequence, countermeasures/management response, risk type/category, and probability and impact. 

These risks include:

  1. Delayed implementation of baseline projects by the government and donors negatively affects EWS project outcomes.

Assumption:    Baseline projects are implemented according to the timeline identified in the PPG phase of the LDCF project, and achieve the desired outcomes and objective.

  1. Installed hydro-meteorological equipment fails because it is vandalized or not maintained.
  2. Assumption:    Communities living nearby installed hydro-meteorological equipment commit to taking active measures to prevent the equipment from being vandalized; and the equipment is adequately maintained by the responsible institution.


Tag: Climate Change Adaptation Natural Disaster Procurement Project and Programme management Risk Management Country Government Donor

6.

1.1.1Planned stakeholder participation

The Project Document specified key project implementation partners and stakeholders and their responsibilities and areas of collaboration within the EWS Project, see Chapter 3.6 Main Stakeholders for details2.6.

Department of Disaster Management Affairs (DoDMA) under the Office of the President and Cabinet was assigned to serve as the lead institution responsible for overall project implementation. Additional main project implementation partners and beneficiaries were Department of Climate Change and Meteorological Services (DCCMS) under the Ministry of Natural Resources, Energy and Mining, and Department of Water Resources (DWR) under the Ministry of Agriculture, Irrigation and Water Development (MoAIWD).

Additional project partners identified in the Project Document included:

  • Department of Surveys under the Ministry of Lands, Housing and Urban Development
  • Ministry of Agriculture and Food Security (MoAFS) – now Ministry of Agriculture, Irrigation and Water Development (MoAIWD)
  • Ministry of Local Government and Rural Development (MoLGRD)
  • Village Civil Protection Committee (VCPC)
  • District Civil Protection Committee (DCPC)
  • Area Civil Protection Committee (ACPC)

Tag: Climate Change Adaptation Disaster risk management Natural Disaster Relevance Implementation Modality Country Government

7.

1.1.1Linkages between the project and other interventions within the sector

There have been many initiatives implemented in Malawi over past decades and especially over last years, even before the EWS Project was launched, related to EWS, and/or supporting development and installation of meteorological and hydrological stations, and recently also implementing EWS, both on a national and community levels.

  • SADC Hydrological Cycle Observing System (HYCOS) Phase 1 Project – installed six hydrological monitoring stations with automatic Data Collection Platforms in the late 1990s
  • Enhancing Community Resilience Project (ECRP, 2011-2015, £21.5 million) funded by the British Department for International Development (DFID), Irish Aid and the Norwegian government, NGO implemented (ECRP and DISCOVER[1]), focused on building community resilience, including low-tech community-based early warning systems
  • Programme Support to Disaster Risk Management (UNDP, PS DRM, 2012-2016, $ 1.35 million) - focused on undertaking capacity development at a national, district and community level to reduce disaster risks and shocks to vulnerable communities
  • Programme Support to Environment and Natural Resources (UNDP, PS ENR, 2013-2016, $1.7 million) provided support to the GoM for mainstreaming environment and natural resources management in policies, development plans and programs at national level
  • Shire River Basin Management Program Phase 1 Project (SRBMP, 2012-2018, ~$125 million) funded by the World Bank, implemented by the MoAIWD, aims to establish coordinated inter-sectoral development planning and coordination mechanisms, undertake the most urgent water related infrastructure investments, prepare additional infrastructure investments, and develop up-scalable systems and methods to rehabilitate sub-catchments and protect existing natural forests, wetlands and biodiversity in the Shire River Basin. The project provides irrigation and flood management infrastructure in the Shire River basin, as well as training and infrastructure for the hydro-meteorological services – this includes the installation of considerable ground- and surface-water measuring equipment in the Shire River Basin in order to provide real-time information to a control center within the DWR.
    • Integrated Flood Risk Management Strategy (IFRMS, 2012-2018, $3.9 million), component of the SRBMP, focused on accurate and timely hydrological measurements. The main aim of IFRMS is to develop a 5-year Action Plan for strategic flood risk management of the Shire River Basin.

Tag: Climate Change Adaptation Disaster risk management Natural Disaster Relevance Bilateral partners International Financial Institutions

8.

1.1.1UNDP comparative advantage

UNDP has a demonstrated administrative and project management capacity to implement EWS projects, it is a neutral GEF implementing agency. UNDP has a substantial in-country and regional expertise and experience from implementing climate change projects in both, Malawi and in other countries of operation in the region and world-wide.

UNDP developed this EWS Project as a part of a broad multi-country program that was designed to implement UNDP-supported similar initiatives on climate information and Early Warning Systems in at least 10 countries in Africa (including Benin, Burkina Faso, Ethiopia, Liberia, Malawi, Sierra Leone, São Tomé & Príncipe, Tanzania, Uganda and Zambia).  This provided additional opportunities for UNDP in facilitating experience and expertise sharing among these UNDP-supported projects in the region.

UNDP has a long-term EWS related country-specific experience providing its Programme Support on Disaster Risk Management and Resilience Building in Malawi.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Global Environment Facility fund Project and Programme management Strategic Positioning

9.

1.1.1      Replication approach and sustainability

The Project Document identified three factors supporting future replication of the EWS Project across the country:

1.     Lessons learned will be disseminated nationally through training programs, the online platform and toolboxes including courses, handbooks and manuals.

2.     Strengthening of capacities among key government stakeholders will enable continued mainstreaming of the use of climate information and early warnings into sectoral planning and decision-making.

3.     The position of the EWS Project in UNDP’s PS DRM will provide the strongest mechanism for replication. Training and capacity building of local communities and technical staff regarding the application of climate information and the response to early warnings will ensure that future local level endeavors within Malawi are climate-resilient.


Tag: Disaster risk management Natural Disaster Sustainability Knowledge management Ownership Country Government Capacity Building

10.

1.1.1Management arrangements

DoDMA was identified as a lead governmental department to serve as a project implementing partner under the NIM modality fully responsible for the overall project management on a daily basis.

The Project Board, consisting of executives of the PS DRM Program and its projects, and National Disaster Preparedness and Relief Committee, was designed to oversee project implementation, including scheduled and ad hoc reviews, and to make key management decisions.


Tag: Disaster risk management Natural Disaster Implementation Modality Oversight Project and Programme management

11.

1.1.1Project implementation and adaptive management

The Project was implemented mostly according to the project strategy and a work plan outlined in the Project Document with one major deviation described below.The Inception Report identified that DCCMS is unable to undertake numerical modelling for weather forecasting due to lack of infrastructure. The major change to the ProDoc outline and the key adaptive management implemented by the EWS Project was that DCCMS decided to improve its weather forecasting capacity by using the COSMO model and ICON model for downscaling.


Tag: Implementation Modality Project and Programme management Results-Based Management

12.

1.1.1Partnerships arrangements

The EWS Project was implemented with project partners identified in the Project Document. Other project partners, including other EWS stakeholders and local NGOs/CSOs, have been invited to participate in project coordination meetings and DRM platform meetings. No additional project partners have been actively involved in direct EWS Project implementation. However, there is a large group of other initiatives with EWS component being implemented in Malawi (see Chapter 3.1.6).


Tag: Relevance Project and Programme management Bilateral partners Coordination

13.

1.1.1Monitoring and evaluation

The Project Document described in detail necessary monitoring framework and evaluation procedures, as required for all UNDP-supported GEF-financed projects.

Specifically, it drafted a Monitoring and Evaluation Work Plan that identified responsible parties for M&E activities, including Inception Workshop and Report, periodic quarterly and annual monitoring and reporting of project progress and performance, annual APR/PIR, Project Board meetings, Midterm Review, Terminal Evaluation, Terminal Lessons Learned Report, and financial audits. For each M&E activity responsible parties have been specified, appropriate indicative budget allocated, and time-frame specified.


Tag: Monitoring and Evaluation Oversight Policies & Procedures

14.

1.1.1Feedback from M&E activities used for adaptive management

The Inception Report of March 2014 summarized the discussion held during the Inception Workshop and no major amendment to the Project Document was proposed. During the discussion need for synergy and convergence between different ongoing programs and projects, need for improved coordination between various stakeholders and government departments, and need to strengthen maintenance capacity were highlighted.

The Inception Report of 2012 prepared at the launch of the project development phase indicated that “initial consultations did not turn up potential private sector clients for the project to interact with, and the UNDP country office pointed out that previous attempts to engage with private sector as a means of ensuring long-term finance for projects have met with little success”. This activity, developing commercial products for sale to private sector, remained in the EWS Project design, leaving securing sufficient post-project funding for O&M at risk. adopt and actualize the recommendations.


Tag: Monitoring and Evaluation Results-Based Management Strategic Positioning Civil Societies and NGOs

15.

1.1.1Financial planning and management

The GEF budget of 3.6 mil USD as of the Project Document is shown in Table 7.

Table 7: Project Budget as of Project Document [USD]

Year

1

2

3

4

Total

 

Outcome 1

286,050

1,171,290

939,600

49,500

2,446,440

68%

Outcome 2

132,050

123,150

236,810

481,550

973,560

27%

Management

47,500

47,500

47,500

37,500

180,000

5%

Total

465,600

1,341,940

1,223,910

568,550

3,600,000

100%

 

13%

37%

34%

16%

100%

 

 

Table 8: Annual EWS Project Expenditures (CDR) [USD] as of July, 2018

 

2014

2015

2016

2017

2018

Total

% of budget

Total

535 515

1 425 310

1 188 339

447 798

0

3 596 962

99.9%

% of total

15%

40%

33%

12%

0%

100%

 

 


Tag: Efficiency Human and Financial resources Operational Efficiency Country Government

16.

1.1.1Co-financing and in-kind contributions

The Project Document specified three sources of co-financing in a total value of 11.3 mil USD, namely the Government of Malawi, UNDP and the UK Department for International Development (DFID), and 30.4 mil USD co-financing (or rather parallel financing) provided by other donors to other projects with EWS component in Malawi.

Total four-year budgets of three implementing governmental agencies (DoDMA, DWR and DCCMS) are indicated in the Project Document as the contribution of the Government of Malawi to the EWS Project implementation – total value 3.838 mil USD.


Tag: Efficiency Human and Financial resources Bilateral partners

17.

1.1.1Management by UNDP and implementing partner

The EWS Project was implemented by regular staff of three governmental departments, namely the DoDMA, DCCMS, and DWR. The Government of Malawi preferred this option rather than project implementation by hired experts and dedicated time-bound Project Implementation Unit, so that the expertise developed through the project implementation period would be more sustainable and will stay with regular staff in their departments. The EWS Project has only hired a Finance and Administrative Assistant.

However, the challenge of this organizational set-up is that the regular staff have their other work obligations and less experience in activities not directly related to their common work, namely with implementing EWS and organization of early warning alerts’ dissemination on a community level to end-users, and in developing risk knowledge and appropriate response capacity.


Tag: Effectiveness Efficiency Human and Financial resources Operational Efficiency Oversight Project and Programme management Country Government

18.

1.1.1Overall results and attainment of objectives

The major result of the EWS Project is that the DCCMS now generates and publishes more reliable, more frequent, and more detailed/region specific weather forecasts, including severe weather forecasts.

The DCCMS managed to significantly improve its weather forecasting capacity thanks to major adaptive management implemented, namely by using (for free) global weather forecasting model COSMO and ICON, and by using the weather input data provided also for free by the German Weather Service DWD. The data from the local met stations are supposed to be used for short-term and seasonal weather forecast, and for downscaling of the COSMO Model forecast to local conditions with the ICON model. In terms of forecast, the DCCMS utilizes this data for verification and information generation on weather (as summarized in forecasts in media); weather analysis as part of weather forecasting (thus, plotting of observed data on the charts to get the prevailing weather systems); and global data exchange (observed data from stations is shared regionally and internationally through the Global Transmission System (GTS) thereby contributing to global weather analysis and forecasting.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Effectiveness

19.

Overview of activities performed and achievements as reported by the EWS Project (some activities being supported by other projects/donors):

Ad 1: Equipment for hydro-meteorological data monitoring

  • 10 AWS installed in 6 of 7 priority districts, of which 9 AWS are operational with hourly reporting via GPRS (and not GSM as originally planned) – located at Mayani, Dzalanyama, Mvera, Chintheche, Chikangawa, Embangweni, Chelinda, Ngabu, Mwimba and Njolomole;
  • Servicing of existing AWSs - change of batteries and data loggers;
  • 30 evaporation pans in all 21 principal meteorological stations and 9 subsidiary/ Agrometeorological Research stations installed, replacement of thermometers and wind measuring instruments in 15 stations, technical stationery, furniture;
  • DCCMS procured 30 computers for data management at station level, 14 laptops;
  • 20 Single Side Band Radios for enhanced communication at all weather stations;
  • 35 basic phones for water gauge readers and 7 smart phones for District Supervisors;
  • Volunteer rainfall stations supported with stationary; and
  • Lightning detection sensors not implemented due to too costly operation and maintenance, and concern that data would not be under Malawian control.

Tag: Climate Change Adaptation Disaster risk management Effectiveness Monitoring and Evaluation

20.

Ad 2: Equipment and communication platforms for weather and EW information dissemination

  • Weather forecasts and EW distributed to civil protection committees (CPC), as well as at district management level.
  • Dissemination of the rainfall seasonal forecast to communities in disaster prone districts of Nkhatabay, Karonga, Rumphi, Nkhotakota, Salima, Dedza, Ntcheu, Balaka, Machinga, Mangochi, Zomba, Phalombe, Nsanje and Chikwawa.
  • DCCMS utilizes the World Meteorological Organizations Common Alerting Protocol (CAP) for dissemination of weather information and warnings through using social media.
  • WhatsApp Weather Chasers group established for weather forecast dissemination.
  • Malawi Weather Facebook account established with 1905 followers – however, the last weather forecast was posted in January 2018, in 2017 only one weather forecast and one seasonal rain forecast posted, more posts available from 2015 and 2016.
  • Mobile application Malawi Zanyengo App was developed, not fully operational yet, only 50+ downloads.
  • Weekly weather updates and alerts are also being accessed through the 321-service provided by mobile service provider Airtel Malawi and Human Network International (HNI).
  • In 2016, five new community based radio stations reported actively weather messages in local languages; 11 national stations are now active in the communication.
  • Coordination protocols and agreements among DCCMS, DWR, DoDMA are being developed by the Shire River Basin Management Program funded by the World Bank.

Tag: Disaster risk management Effectiveness Communication Monitoring and Evaluation

21.

Ad 3: Trainings and capacity strengthening

  • 17 meteorologists trained in Linux operating system and weather modeling using the COSMO-model.
  • 5 DCCMS officers trained in a two-year advanced meteorological forecasting course (WMO CLASS II) at Institute of Meteorological Training in Nairobi, Kenya.
  • 1 DCCMS officer was supported to undergo a post-graduate diploma studies in Operational Hydrology specializing in Early Warning at the University of Nairobi, Kenya.
  • 2 DCCMS officers underwent a training course on Nowcasting Techniques on Thunderstorms and Severe Convections in China.
  • DCCMS trained 87 meteorological observers, technicians and officers in data management using CLIMSOFT data management software.
  • SEBA instrument training and troubleshooting course on DWR automatic river gouge stations in Germany.
  • Training in HYDSTRA Hydrological Data Management Information system.
  • 25 Water professionals and technicians trained in Integrated Water Resources Management.
  • 590 community members (302 women, 288 men) were trained in use of seasonal forecasts.
  • Villages Civil Protection Committees (VCPC) in 11 districts received training in understanding and interpretation of weather messages and forecasts, on the benefits of early warning systems and the role that communities can play in disaster risk management.
  • Documentary on flooding that occurred in January, 2016 was produced, in order to capture the effects of El Nino weather condition.
  • DCCMS has also developed assorted early warning information, education and communication (IEC) materials under the project that have been distributed to district climate information centers and other stakeholders.
  • Environmental Sciences and Management Department (ESMD) of the Lilongwe University of Agriculture and Natural Resources was engaged to develop the training package and toolkit.

Project objective and outcome level results and rating are summarized in Table 10: Project results and achievements as per LogFrame targets below.

 


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Effectiveness Capacity Building

22.

Table 10: Project results and achievements as per LogFrame targets

Rating refers to achievements of targets supported by the EWS Project. Achievements of other projects are not included in the EWS Project LogFrame rating.


Tag: Effectiveness Project and Programme management

23.

1.1.1Relevance

The part of the EWS Project objective that concerns “strengthening hydrological monitoring capabilities, early warning systems and available information for responding to extreme weather and planning adaptation to climate change in Malawi” is highly relevant to national policies and development priorities in Malawi, as well as to GEF Operational Programme and GEF and UNDP strategic priorities over the whole EWS Project implementation period. See Chapter 3.1.1 Project Relevance for details.

The first part of the EWS Project objective “strengthening the weather and climate monitoring capabilities” is rather an assumption that these capabilities need to be strengthened for improved weather forecasting and strengthened EWS.

Utilizing internet based global weather forecasting model and globally available data for improved and region-specific weather forecasting in Malawi proved that the assumption of strengthened weather monitoring is not valid for improved weather forecasting capacity.

Project relevance is rated Relevant.

This reflects relevance of the project objective, rather than relevance of specific project strategy outlined in the Project Document that was not designed to be appropriate/affordable for Malawi.


Tag: Climate Change Adaptation Disaster risk management Relevance Global Environment Facility fund Capacity Building

24.

1.1.1Effectiveness of project implementation

Effectiveness of project implementation evaluates an extent to which an objective has been achieved.

The project objective has been only partially achieved. The implicit project objective of strengthened weather forecasting capacity was achieved.  Severe weather forecasting based on global weather forecasting model has been strengthened. EWS have been partially strengthened: technical equipment to facilitate early warning dissemination has been provided. However, effective implementation of EWS and community-based delivery of severe weather warnings was achieved only partially. Local manual reading of upstream river water levels is only partially operational, it is not read and reported regularly, since no reimbursement is provided since September 2017 (i.e. including the last rainy season). Planning adaptation to climate change has been partially achieved (National DRM Policy adopted, but only 2 of 7 target district development plans adopted). Little evidence in achieving improved risk knowledge and appropriate response capacity was found. All GEF project budget resources have been fully spent.

Compared to designed project results and EWS Project budget, the effectiveness of project implementation is rated Moderately Unsatisfactory.

 


Tag: Effectiveness Implementation Modality Project and Programme management Capacity Building

25.

1.1.1Efficiency (cost-effectiveness) of project implementation

UNDP defines project efficiency (cost-effectiveness or efficacy) as an extent to which results have been delivered with the least costly resources possible.

One of 10 installed automatic weather stations is not operational because it has no mobile signal, 2 of 12 installed automatic river water level stations are not operational, data from rehabilitated manual weather and water level stations are not fully collected due to insufficient O&M budget. The equipment installed has been partially not operational even before project end due to lack of O&M financing, 10% of AWS were installed in a location without a mobile signal, 20% of AHS are not operational because they were damaged by floods. Project funds spent on purchase and rehabilitation of meteorological stations and some other equipment were not effectively spent.

The cost-effectiveness/efficiency of project implementation is rated Unsatisfactory.


Tag: Effectiveness Efficiency Human and Financial resources Operational Efficiency

26.

1.1.1Country ownership

The EWS Project is a typical donor-driven project. It was developed within a framework of a broad multi-country program that was designed to implement similar initiatives on climate information and Early Warning Systems in at least 10 countries in Africa (including Benin, Burkina Faso, Ethiopia, Liberia, Malawi, Sierra Leone, São Tomé & Príncipe, Tanzania, Uganda and Zambia).

Formally, the EWS Project is fully owned by the country. It was implemented by regular staff of governmental departments of DoDMA, DCCMS and DWR, new and first National DRM Policy was developed and adopted, EWS is prioritized in national policies developed with donors’ support in response to recent tragic floods and droughts that resulted in food shortages and humanitarian crisis, EWS projects have been replicated across the country.

However, the good formal country ownership has not materialized in providing sufficient O&M funding in order to keep all installed equipment operational. With weather stations, it is understandable, since it does not undermine the country’s capacity in improved weather forecasting. In case of hydro stations, the lack of funding for water level readers undermines the capacity in the country to issue site-specific flood warnings. The volume of funding is negligible compared to the EWS Project budget: it is an equivalent of 10 USD per month and one reader/hydro manual station, however, in total it represents ca 10%[1] of the DWR budget. Other community-based EWS projects implemented by NGOs in Malawi succeeded to motivate local communities to volunteer in river water level readings, including upstream communities not affected by floods, and thus the operation and readiness of EWS does not depend on O&M funding.

 


[1] Need to revise/update after an information on the DWR budget will be provided.


Tag: Climate Change Adaptation Disaster Risk Reduction Natural Disaster Ownership Country Government Donor

27.

1.1.1Prospects of sustainability

1.1.1.1Financial risks

Financial risk, particularly the typical risk of no/low post-project financing for operation and maintenance of installed equipment, is the main risk for sustainability of the EWS Project. Unfortunately, this risk fully materialized already during the project implementation period, when data from manual stations are not fully read due to a lack of (negligible) funding needed for O&M, such as lack of paper for temperature and humidity mechanical recording, water bills not paid and thus water supply was interrupted and thus no water was available to refill evaporation pans (few liters a day maximum), no funding is available since September 2017 for reimbursement of personnel reading river water level data (10 USD per month per person), etc.

The proposed strategy to increase revenues that would help to cover O&M costs failed and no commercial weather services were developed for sale.

Prospects of financial resources sustainability is rated Unlikely.


Tag: Climate Change Adaptation Sustainability Project and Programme management

28.

1.1.1.1Socio-Political Risks

There were no significant social, nor political risks to project sustainability identified. However, the government ownership of the EWS Project, although formally declared, did not materialize in prioritizing EWS through provision of sufficient funding for operation and maintenance. The country depends and relies on development partners for disaster management funding.

The socio-political/economic sustainability is rated Moderately Likely.

 


Tag: Risk Management Country Government Sustainability

29.

1.1.1.1Institutional Framework and Governance Risks

No institutional nor governance risks have been identified that might jeopardize project sustainability.

Institutional framework and governance sustainability is rated Likely.


Tag: Sustainability Country Government

30.

1.1.1.1Environmental Risks

One automatic hydro-station was flooded and became inoperable and some manual river water level stations were damaged by floods (and vandalism). No other environmental risks to project sustainability were identified.

Environmental sustainability is rated Moderately Likely.

Overall prospects of sustainability of delivered project results are rated Unlikely.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Sustainability

31.

1.1.1Catalytic Role

The EWS Project was not the only one, nor the first project targeting EWS implemented in Malawi. There has been a good practice gained and replicated in Malawi in community-based EWS implemented by NGOs before this UNDP-supported GEF-financed EWS Project was launched. This EWS Project is unique by supporting significant quality improvement of weather forecasting capacity of DCCMS. The EWS Project demonstrated some catalytic role only in providing improved weather forecasts.


Tag: Effectiveness Sustainability Strategic Positioning

32.

1.1.1Project Impact

Project impact evaluates impact on environmental status improvement and environmental stress reduction. The EWS Project was not designed with the aim to address environmental status improvement, nor environmental stress reduction. It was rather designed to reduce negative impacts of severe weather conditions, namely floods, strong winds, and droughts on human beings, casualties, and food security.

Despite the fact that recorded casualties have changed year-to-year, there are no statistically relevant data for any conclusions due to short period of data available after improved weather forecasts and EWS have been implemented.

The EWS Project had significant impact on improved quality and reliability of weather forecasts, including forecasting of severe weather conditions, as reported by all stakeholders, governmental officers, department experts, as well as by general public – end-users of weather forecasts and EWS (Outcome 1). The impact of delivered achievements in Outcome 2 by the EWS Project itself, without achievements delivered by other projects, is much lower.

Overall project impact is rated at the middle of the three level scale, i.e. Minimal in UNDP/GEF scale (more appropriate wording would be Moderate).

 


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Effectiveness Impact Country Government

33.

The additional risk identified in the LogFrame is:

  1. Alerts and warnings required by communities are not feasible to produce due to scientific or technological barriers.

Assumption:    The most up to date technology and scientific approaches and advances are feasible and appropriate for meeting the LDCF project needs. The level of error for forecasting is within the minimum thresholds appropriate for the LDCF project activities.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Relevance Project and Programme management Risk Management Country Government

34.
  • Southern African Regional Climate Outlook Forum (SARCOF), which is a Southern African Development Community (SADC) coordinated platform on which Malawi participates in the analysis of regional weather forecasting with SADC.
  • Group on Earth Observations’ (GEO) AfriGEOSS initiative
  • African Monitoring of the Environment for Sustainable Development (AMESD) and Monitoring of Environment and Security in Africa (MESA)
  • WMO’s Global Framework Climate Services (GFCS) initiative
  • The Malawi Red Cross Climate Change Project, funded by the Finish Red Cross, provided mobile weather stations
  • Disaster Preparedness ECHO (DIPECHO), launched in 2008, funded by the European Community Humanitarian Office (ECHO), implemented community-based EWS in Chikwawa, Salima and Nsanje districts by international NGOs. It includes simple river level gauges in upstream villages, which are read by community members regularly. Warnings are disseminated to downstream village civil protection committees (CPCs) using mobile phones, megaphones, whistles and community flags. This low-technology warning system has proved to be fairly effective, and the methodology has been adopted by the ECRP.
  • Africa Adaptation Programme (AAP, JICA funded), implemented by the UNDP and FAO, established district climate change information centers and implemented community-identified adaptation measures in 7 districts, and a community-based flood warning system in Salima district.
  • Southern Africa Flash Flood Guidance (SARFFG) system promotes exchange of information on flash floods between regional meteorological networks. The South African Weather Service (SAWS) is providing information on flash flood warnings. This information is integrated into DCCMS’s early warning systems.

 


[1] Developing Innovative Solutions to Overcome Vulnerability through Enhanced Resilience


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Relevance Bilateral partners International Financial Institutions

Recommendations
1

When designing new projects especially in low-income countries, always realistically enumerate specific post-project incremental costs, such as operational and maintenance costs, needed for ensuring project sustainability and for reliable operation of installed equipment, and clearly identify credible sources of long-term post-project funding. Only project proposals with enumerated and secured post-project financing in a long-term should be approved.

2

UNDP and GoM are encouraged to motivate international donors to support development of a single coordinated long-term DRM/EWS national implementation plan/action plan that would increase ownership and coordination responsibility of the GoM and allow international donors to finance implementation of specific phases or areas of the DRM/EWS action plan.

3

In any follow-up to this EWS Project, strengthen EWS coordination horizontally and vertically, in order to improve effectiveness of early warning dissemination and to support the goal of prospective development of an integrated nation-wide early warning system. Utilize the experience of NGOs/CSOs in Malawi and consider their active engagement in organizing early warning dissemination within communities, and in developing community-specific concrete response capability and risk knowledge among population.

4

When designing new EWS projects especially in low-income countries, always consider the least-cost option, i.e. utilization of free localized weather forecasts based on different global and regional numerical weather forecasting models that are available from various weather services and numerous internet platforms (such as windy.com, accuweather.com, yr.no, wunderground.com, and many more). Analyze benefits/value added of downscaling weather forecasts locally, and costs needed for hardware and software infrastructure upgrades.

1. Recommendation:

When designing new projects especially in low-income countries, always realistically enumerate specific post-project incremental costs, such as operational and maintenance costs, needed for ensuring project sustainability and for reliable operation of installed equipment, and clearly identify credible sources of long-term post-project funding. Only project proposals with enumerated and secured post-project financing in a long-term should be approved.

Management Response: [Added: 2019/01/05] [Last Updated: 2021/01/04]

UNDP agrees that project results should be sustainable, and one of the indirect results from this project was increased financing for Department of Climate Change and Meteorological Services which the evaluator did not acknowledge. However, the increased financing did not cover operation and maintenance costs as government did not establish a business case for beneficiaries of the established early warning to pay for some service and generate revenue for operating maintaining the systems. Given that all projects have a specific timeline, UNDP Malawi is of the position that projects which involve new innovations and procurement of equipment should have the operation and maintenance costs taken over by the host government, preferably by clearly establishing a business case for the private sector so that interventions and results are fully owned and sustained.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Future similar projects will have deliberate commitments from the host government at inception stage on post-project operation and maintenance costs
[Added: 2019/01/05]
Resilience and Sustainable Growth Portfolio 2018/12 Completed This is an undertaking which will be monitored throughout the 2019-2023 Country P rogramme.
2. Recommendation:

UNDP and GoM are encouraged to motivate international donors to support development of a single coordinated long-term DRM/EWS national implementation plan/action plan that would increase ownership and coordination responsibility of the GoM and allow international donors to finance implementation of specific phases or areas of the DRM/EWS action plan.

Management Response: [Added: 2019/01/05] [Last Updated: 2021/01/04]

UNDP Malawi agrees with the recommendation as it would improve coordination among different players in the country. While UNDP does not have control over government decisions, efforts will be made to support development of protocols for coordinating DRM/EWS action plan. At the time of the evaluation, Malawi government had completed a National Resilience Strategy (NRS) which is aiming coordinating DRM and EWS initiatives, UNDP will support implementation of the NRS.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Support implementation of the National Resilience Strategy
[Added: 2019/01/05]
Resilience and Sustainable Growth Portfolio. 2018/12 Completed The Country Office is prepared draft project document on disaster risk reduction which will support the operationalization of National Resilienc Strategy.
3. Recommendation:

In any follow-up to this EWS Project, strengthen EWS coordination horizontally and vertically, in order to improve effectiveness of early warning dissemination and to support the goal of prospective development of an integrated nation-wide early warning system. Utilize the experience of NGOs/CSOs in Malawi and consider their active engagement in organizing early warning dissemination within communities, and in developing community-specific concrete response capability and risk knowledge among population.

Management Response: [Added: 2019/01/05] [Last Updated: 2021/01/04]

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Support development of National Guidelines for Community Based Early Warning Systems
[Added: 2019/01/05]
Resilience and Sustainable Growth 2018/12 Completed UNDP has made concrete plans to support development of guidelines by March, 2019.
4. Recommendation:

When designing new EWS projects especially in low-income countries, always consider the least-cost option, i.e. utilization of free localized weather forecasts based on different global and regional numerical weather forecasting models that are available from various weather services and numerous internet platforms (such as windy.com, accuweather.com, yr.no, wunderground.com, and many more). Analyze benefits/value added of downscaling weather forecasts locally, and costs needed for hardware and software infrastructure upgrades.

Management Response: [Added: 2020/12/20] [Last Updated: 2021/01/04]

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org