Strengthening Climate information and Early Warning System Terminal Evaluation

Report Cover Image
Evaluation Plan:
2017-2020, Ethiopia
Evaluation Type:
Final Project
Planned End Date:
08/2018
Completion Date:
09/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
50,000

Share

Document Type Language Size Status Downloads
Download document TOR-SCIEWP.docx tor English 45.18 KB Posted 491
Download document Terminal Evaluation_SCIEWP_FINAL_SUBMITTED 16 August.docx report English 579.96 KB Posted 474
Title Strengthening Climate information and Early Warning System Terminal Evaluation
Atlas Project Number: 00073414
Evaluation Plan: 2017-2020, Ethiopia
Evaluation Type: Final Project
Status: Completed
Completion Date: 09/2018
Planned End Date: 08/2018
Management Response: Yes
UNDP Signature Solution:
  • 1. Resilience
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.1.1 Capacities developed across the whole of government to integrate the 2030 Agenda, the Paris Agreement and other international agreements in development plans and budgets, and to analyse progress towards the SDGs, using innovative and data-driven solutions
SDG Goal
  • Goal 13. Take urgent action to combat climate change and its impacts
SDG Target
  • 13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries
Evaluation Budget(US $): 50,000
Source of Funding: GEF
Evaluation Expenditure(US $): 36,475
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
David Wright International Consultant
Gizachew Getaneh local Consultant
GEF Evaluation: Yes
GEF Project Title: Strengthening Climate information and Early Warning System Terminal Evaluation
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: FSP
GEF Phase: GEF-5
GEF Project ID: 4992
PIMS Number: 5095
Key Stakeholders: National Meteological Agency, Ministry of water irrigation and energy, DRMFSS
Countries: ETHIOPIA
Lessons
1.

0.1.1Lessons from other relevant projects incorporated into project design

This project is one of 10 country-led LDCF-funded projects in Africa focused on strengthening climate information and early warning systems for climate resilient development and adaptation. Other countries receiving support are: Benin, Burkina Faso, Liberia, Sierra Leone, Sao Time and Principe, Uganda, Tanzania, Malawi and Zambia. The design of this project benefitted from the experience of a regional team of UNDP technical specialists with thematic expertise in areas including: meteorological, climate and hydrological observation networks, weather, climate and hydrological forecasting, data management and archiving, disaster management, agricultural advisories, flood alerts, communication protocols and standard operating procedures, mobile communications, private sector engagement, and development of weather and climate services. The project also benefitted from UNDP experience managing other climate information and early warning projects in Africa, and Asia-Pacific, funded through LDCF, SCCF and bi-lateral funds.


Findings
1.

0.1Project Design/Formulation

0.1.1Analysis of Project Results Framework

Logic

The Project Results Framework (PRF) identifies the project’s objective and two complimentary outcomes. The project objective is strongly aligned with UNDP Country Programme Outcomes and GEF Climate Change Adaptation Focal Area Objectives CCA-2 (Increase adaptive Capacity to respond to the impacts of climate change, including variability, at local, national, regional and global level) and CCA-3 (Promote transfer and adoption of adaptation technology).

While the PRF included in the Project Identification Form (PIF) includes the project’s expected outputs, these are omitted from the PRF in the Prodoc. The expected outputs are identified under Section 2.4 of the Prodoc however, no output level indicators are identified or described. The omission of outputs from the PRF, and lack of any output level indicators, compromises the presentation of project logic and limits project capacity to effectively monitor and report progress, and to identify potential adaptive management requirements.


Tag: Climate Change Adaptation Relevance Global Environment Facility fund Monitoring and Evaluation Programme/Project Design Capacity Building

2.

Outcome 1: Enhanced capacity of NMA and HWQD to monitor extreme weather and climate change

Progress towards outcome 1 is measured using two indicators. The first indicator “Percentage of national coverage of weather/climate and hydrological monitoring infrastructure” is erroneously articulated. The target “Increase in 24% national coverage to take steps in achieving NMAs optimal monitoring arrangements as defined in feasibility studies” is simply the target percentage increase of functioning manual meteorological stations (1240) against the baseline (1000).  This is not representative of national coverage, it represents only the target number of manual meteorological stations to be rehabilitated by the project. Secondly, the target only measures the percentage increase in manual meteorological stations, it does not measure the percentage increase for AWS (baseline = 70; target = 110) or hydrological stations with telemetry (baseline = 4; target = 64) which are major focuses of the project. Based on the identified baselines and targets, the target percentage increase for these stations would be 57% and 1,500% respectively. The second indicator “Frequency and timeliness of climate related data availability” form a baseline of monthly to daily is satisfactory (though the baseline reported in the Prodoc is four-monthly for manual hydrological stations


Tag: Climate Change Adaptation Monitoring and Evaluation Capacity Building

3.

Outcome 2: Percentage of population with access to improved climate information and flood and drought warnings (disaggregated by gender)

Progress towards outcome 2 is measured using three indicators. However, it is unclear how the project would measure the first indicator “Percentage of population with access to improved climate information and flood and drought warnings” since no source of verification is identified. The second and third indicators “Development frameworks that integrate climate information in the formulation [sic.]and “Sector-specific EW products and strategies that integrate climate risk” have appropriate targets and sources of verification.


Tag: Climate Change Adaptation Natural Disaster Relevance Monitoring and Evaluation

4.

0.1.1Assumptions and Risks

The project’s key risks and assumptions were identified during bilateral consultations with project stakeholders during the project preparation phase. Risks are presented in the risk log under Annex 1 of the Prodoc. Assumptions are identified in the Prodoc but are not recorded in the PRF.  The risk and assumptions identified supported the development of activities and planned outputs. For example, “Unavailability of requisite human resources and data” and “Limited capacity within relevant ministries/insufficient qualified human capacity” ensured the project design had a strong focus on both institutional and technical capacity for planning, designing and implementing local level adaptation actions.


Tag: Relevance Communication Human and Financial resources Programme/Project Design Project and Programme management Risk Management

5.

Lessons from other relevant projects incorporated into project design

This project is one of 10 country-led LDCF-funded projects in Africa focused on strengthening climate information and early warning systems for climate resilient development and adaptation. Other countries receiving support are: Benin, Burkina Faso, Liberia, Sierra Leone, Sao Time and Principe, Uganda, Tanzania, Malawi and Zambia. The design of this project benefitted from the experience of a regional team of UNDP technical specialists with thematic expertise in areas including: meteorological, climate and hydrological observation networks, weather, climate and hydrological forecasting, data management and archiving, disaster management, agricultural advisories, flood alerts, communication protocols and standard operating procedures, mobile communications, private sector engagement, and development of weather and climate services. The project also benefitted from UNDP experience managing other climate information and early warning projects in Africa, and Asia-Pacific, funded through LDCF, SCCF and bi-lateral funds.


Tag: Climate Change Adaptation Natural Disaster Relevance Sustainability Knowledge management

6.

Planned stakeholder participation

During the project preparation phase, three stakeholder consultations were conducted at inception, mid-way through project design and at validation. These consultations included government agencies, national and international NGOs and UN agencies. National government agencies were represented by the Federal Environmental Protection Agency (EPA), Ministry of Water, Irrigation and Electricity, Ministry of Agriculture, NMA, HWQD, NDRMC, Ethiopian Institute of Agricultural Research (EIAR), regional NMA and HWQD branch offices and regional, zonal, and woreda level DRM experts. NGOs were represented by Oxfam America and the Ethiopian Climate Change Forum, and UN Agencies were represented by UNDP, World Food Programme (WFP) and African Climate Policy Centre (ACPC). Additionally, consultations were held with DFID, USAID and the World Bank.  The Prodoc states that the Ethiopian Coffee Farmers Association and University of Addis Ababa were consulted however, the Stakeholder Baseline Analysis, presented in Annex VII of the Prodoc, does not identify consultation with these stakeholders (though the University of Addis Adaba is included in the Stakeholder Involvement Plan).


Tag: Relevance Human and Financial resources Bilateral partners Country Government UN Agencies Civil Societies and NGOs

7.

Replication approach

Replication was a core consideration in the design of this project. The climate information and early warning capacity needs, both in terms of infrastructure and human resources, in Ethiopia are significantly greater than the resources available through this project alone and this project may act as a demonstration for other projects to replicate.For example, this project planned to procure and install 40 AWS. However, the national target, as identified in NMAs Second Growth and Transformation Plan (2015-2019) is 700. At the time of this evaluation, approximately 250 AWS have been procured and installed with the support of various agencies (UNDP met its target and supported the installation of 40 AWS). Similarly, this project planned to procure and install 10 hydrological monitoring stations and rehabilitate a further 50 with telemetry, archiving and data processing facilities.


Tag: Climate Change Adaptation Relevance Human and Financial resources Knowledge management Capacity Building

8.

UNDP comparative advantage

UNDPs comparative advantage lies in the agencies deep multi-level and multi-sectoral commitment to advance the disaster risk, early warning and environmental agenda in Ethiopia. UNDP supported Ethiopia to prepare its First National Communication, in response to its commitments to UNFCCC, and its National Adaptation Programme of Action (NAPA). Subsequently, it has implemented projects to build disaster risk management and early recovery capacity, improving resilience of rural communities to climate shocks and contributing to safeguarding of food security.

This project is in line with UNDP Ethiopia’s commitments to support adaptation to climate change and will contribute to UNDPs global climate adaptation strategic outcome: “Strengthened capacity of developing countries to mainstream climate change adaptation policies into national development plans”. Furthermore, the overarching strategic approach of the UNDP Country Programme Document (CPD 2012-2015) is to strengthen capacities of national actors, systems and institutions, through targeted and catalytic interventions that accelerate broad-based development and safeguard development gains against shocks.

UNDPs comparative advantage is further demonstrated by this project’s links to other country-led LDCF-funded projects in Africa focused on strengthening climate information and early warning systems for climate resilient development and adaptation, and significant experience implementing GEF Climate Change Adaptation Focal Area projects globally.


Tag: Climate Change Adaptation Disaster Risk Reduction Relevance Strategic Positioning Capacity Building

9.

Linkages between project and other interventions within the sector

The Prodoc identifies over USD 33M of national programmes/projects that address baseline related problems. The project identified these initiatives as co-financing however, progress and expenditure of these projects was not tracked. The project was designed to build on these initiatives and implement complimentary activities:

  • UNDP: Strengthening Capacities for Ethiopia’s Disaster Risk Management System (US$ 13M), which aims to reduce disaster risk and impact of disasters through the establishment of a comprehensive and integrated disaster risk management system
  • USAID: Pastoralists Resiliency Improvement and Market Expansion (PRIME) project (USD 10M), which aims to identify and implement actions to reduce vulnerability to climate change
  • USAID: Enhanced Livelihoods Application through the Livelihood Integration Unit (US$ 6.8M), which aims to support the Ethiopian Ministry of Agriculture and the NDRMC to strengthen early warning systems
  • ACPC: Support to the National Meteorological Agency (NMA) of Ethiopia in Capacity Building of Climate Monitoring and Upgrading and Expansion of Stream flow Observing and Data Management Systems in Ethiopia (total US$ 1.2M), which aim to develop NMAs data management capacity, install automatic hydrological observation instruments and develop HWQDs data management capacity
  • UNFAO: Strengthening Capacity for Climate Change Adaptation in Land and Water Management with focus on Sustainable Land Management in Ethiopia (US$ 550,000), which aims to reduce the impact of climate change and variability on smallholder farmers within the Awash Basin
  • WFP: Establishment of Automated Weather Stations (AWS) to Improve Timely Provision of Weather Data (US$ 362,700)
  • WFP: Strengthening Early Warning Systems and Disaster Preparedness (US$ 252,000)
  • WFP: Improving Automated Hydrological Data for Flood Model Development (US$ 115,000)

 


Tag: Relevance Programme Synergy

10.

Management arrangements

Management arrangements identified in the Prodoc were as per UNDPs National Implementation Modality (NIM). The Implementing Partner (IP) for the project was the NMA. The Responsible Parties were HWQD and NDRMC, and their regional offices nationwide.

The proposed Project Board comprised an Executive (Chair), Senior Supplier and Senior Beneficiary. The Executive role was assigned to MoFEDs UN Agencies and Regional Economic Cooperation Director, as the representative of the Government Cooperating Agency, with UNDP proposed as Co-Chair. The Senior Supplier role, which represents the interests of the funder, was assigned to the Head of Climate Change Resilience, Green Growth Unit, UNDP Ethiopia. The Senior Beneficiary role, which represents the interests of project beneficiaries, was assigned to the Director General of NMA.

Project assurance (implementation oversight) was assigned to the UNDP Ethiopia Climate Change and Vulnerabilities team, supported at the regional and global level by UNDP-GEF. Day-to-day project management was assigned to a Project Manager (with appropriate support), with responsibility to plan, coordinate and monitor activities delivered by the IP and responsible Parties. The proposed management arrangements are outlined in figure 1.


Tag: Relevance Human and Financial resources Implementation Modality Project and Programme management Quality Assurance

11.

Project Implementation

Adaptive management 

The project was designed with two outcomes and eleven outputs (six contributing to outcome 1 and five contributing to outcome 2). However, Annual Work Plans (AWPs) identify a third outcome (note that AWPs refer to outcomes as outputs and outputs as activities): Effective and efficient project management and knowledge management, communication, stakeholder engagement. This outcome included three outputs (activities): 1) an integrated project management system established to enable coordination, synergy and efficient and effective management of the project and related initiatives; 2) establish project governance structure; 3) put knowledge management and communication strategy in place. The overall target for this outcome was to put in place an integrated Project Management Unit, stakeholder platform, and communication and knowledge management system. While this outcome, and associated outputs, was included in all AWPs, progress was not reported in PIRs, where no reference is made to this outcome. The project did not deliver output 1 — no PMU was established and operationalised. This resulted in limited progress towards output 3 — knowledge management and communication. Progress towards output 2 included the establishment of a Project Board (Project Steering Committee) however, although the Project Steering Committee was established, and meetings were regularly held during the first half of project implementation, stakeholder participation was low and the last meeting was held in August 2016.


Tag: Relevance Programme/Project Design Project and Programme management Results-Based Management

12.

Partnership arrangements 

As planned, a Project Board (Project Steering Committee) was established to provide direction to the IP. Specifically, the planned responsibilities for the Project Steering Committee were to:

  • Supervise project activities through monitoring progress and approving annual reports
  • Review and approve AWPs, financial plans and reports
  • Provide strategic advice to the IP and Responsible Parties to ensure integration of project activities with national and sub-national sustainable development and climate resilience objectives
  • Ensure inter-agency coordination and cross-sectoral dissemination of strategic findings
  • Ensure full participation of stakeholders in project activities
  • Assist with organisation of project reviews and contracting consultancies under technical assistance
  • Provide guidance to the Project Manager

However, the Project Steering Committee was not well attended by its appointed stakeholders — several members never attended, two members interviewed by the evaluation team could not recollect what the project was about, and one also stated that they were unaware of their responsibilities since no ToR was circulated.  The Steering Committee met once in 2014, twice in 2015 and twice in 2016. However, there have been no meetings since August 2016, shortly after the Mid-term Review.


Tag: Relevance Human and Financial resources Oversight Project and Programme management

13.

The project is supported by a regional programme implemented by UNDP and funded by LDCF: Climate Information for Resilient Development in Africa (CIRDA). CIRDA supports 10 African countries to strengthen climate information and early warning capacity and provides assistance that would not be financially viable under national projects. The 2015 PIR highlights significant support from CIRDA including:

  • Workshop: Inception (14-15 April 2014, Ethiopia): to understand country-specific needs and expectations
  • Workshop: Building a Sustainable Climate Change Adaptation and Economic Development Plan (3-5 March 2015, Uganda): to discuss value-added weather and climate services through public-private partnerships to generate revenue and improve public service delivery
  • Workshop: A Systems Approach to Designing, Implementing and Utilising Observing Networks (14-16 October 2014, Tanzania): to provide a systems perspective on weather and climate observing networks
  • Development of Long Term Agreements (LTAs) with hydromet observational equipment suppliers and services to streamline in-country procurement, reducing prices and expediting delivery lead times
  • Facilitation of knowledge-sharing between beneficiary countries via an Information Sharing Platform (website and blog)
  • Support to develop public-private sector strategies to pay for tailored climate information products

Despite significant support reported in the 2015 PIR, on-going CIRDA activities are not reported in later years.


Tag: Climate Change Adaptation Relevance Communication Knowledge management Bilateral partners Capacity Building

14.

Feedback from M&E activities used for adaptive management

Effective adaptive management procedures were not established by the project — the Project Steering Committee was not effective, no dedicated Project management Unit (PMU) was established and the PRF was incapable of measuring output level progress. Furthermore, the GEF Focal Area Tracking Tool was not utilised. Whilst there is no evidence of M&E reported in the PIRs, UNDP did provide a management response to the Mid-term Review. However, many recommendations were not implemented. Further details are provided in the following sections.


Tag: Relevance Monitoring and Evaluation Project and Programme management

15.

Project Finance

The Prodoc identified over USD 33M of co-financing. However, these funds were the sum of identified project finance committed to address project baseline related issues. Expenditure against these projects was not monitored and, during the evaluation missions, the evaluation team confirmed with UNDP that no co-finance was secured. The IP and Responsible Partners stated that they had committed in-kind support in respect of staff time and facilities. Whilst the AWPs record an annual Government in-kind budget of USD 150,000, interviews with relevant personnel did not clarify how this budget was calculated and no expenditure records were maintained.


Tag: Efficiency Human and Financial resources Operational Efficiency Project and Programme management

16.

The audit concluded that the CDR statements fairly present the total expenses of USD 712,881.07 (Government: USD 679,166.07; UNDP direct payments: 33,715.00) incurred by the project in accordance with agreed accounting policies and were: i) in conformity with the approved project budget; ii) for the approved purposes of the project; iii) in compliance with the relevant UNDP regulations and rules, policies and procedures; and, iv) supported by properly approved vouchers and other supporting documents.  Furthermore, the audit concluded that the statements of assets and cash position present fairly, in all material respects, the balances of assets, equipment, cash and bank balance of the project.

The audit also covered the overall management of the project’s implementation, monitoring and supervision.  In terms of budget utilisation, project progress and timeliness, the audit concluded that performance was satisfactory, since NMA had utilised 100% of funds advanced over the course of the audit period. In terms of internal control, the audit stated that MoFEDs Internal Audit Department monitored project expenditure quarterly and concluded this showed commitment of the Government, IP and Responsible Parties to improve internal control. However, the audit also noted that project fixed assets were not accounted at year end and were not affixed with asset identification numbers.


Tag: Efficiency Human and Financial resources Operational Efficiency Oversight Project and Programme management

17.

Whilst the independent financial audit reported positively overall, in 2015 the project experienced major financial management issues, leading to significant implementation delays at HWQD. As IP, NMA was responsible for disbursing funds to the Responsible Parties, following advancement of funds from UNDP. This process proved so inefficient that the delay in HWQD receiving project funds resulted in almost all activities identified in the AWP for 2015 being rescheduled for 2016. Mitigating this inefficiency, disbursement of funds to the Responsible Parties for activities identified in the AWPs for 2016 and 2017 was advanced directly from UNDP.


Tag: Efficiency Human and Financial resources Operational Efficiency Project and Programme management

18.

This showed a significant underspend of USD 365,883.43 against outcome 2 and a significant overspend of USD 237,798.50 against outcome 3.  The overspend against outcome 3 is initially surprising, since no PMU was established. However, UNDP commented that savings incurred by not establishing a PMU, coupled with an underspend against outcome 2, were used to procure a High-Performance Computer costing USD 351,000. It is unclear why this procurement cost has been recorded under outcome 3. Furthermore, the underspend against outcome 2 is surprising. In 2016, HWQD reported that the budget for awareness raising and training of 60 manual hydrology station observers was insufficient and that they would need to identify alternative financing.

In summary, it was difficult to evaluate financial control. The budget approved in the Prodoc was greater than the budget realised; the financial audit covered only two years, and then, only 25% of total expenditure during that time; financial reporting was inconsistent between PIRs; and CDRs do not breakdown expenditure by outcome, output and activity.

As of 3 May 2018, total project expenditure was USD 4,486,635.18, leaving a surplus of USD 13,364.82. However, TE costs are yet to be disbursed.  


Tag: Efficiency Human and Financial resources Operational Efficiency Project and Programme management

19.

Monitoring and evaluation: design at entry and implementation

Monitoring and Evaluation design at entry is rated as: MODERATELY UNSATISFACTORYThe M&E plan outlined in the Prodoc included provision for a Project Inception Workshop, annual APRs/PIRs, Mid-term Review, Terminal Evaluation and periodic site visits. It also stated that the project would identify, and participate in, scientific, policy-based or other relevant networks which may benefit the project through lessons learned; and would interact with other parallel projects to facilitate information flows.


Tag: Efficiency Human and Financial resources Monitoring and Evaluation Operational Efficiency Project and Programme management

20.

Monitoring and Evaluation implementation is rated as: UNSATISFACTORY

M&E during project implementation was particularly weak. As outlined in Section 3.2.2, the Project Board (Project Steering Committee) was not well attended by its appointed stakeholders — several members never attended and two members interviewed by the evaluation team could not recollect what the project was about. Whilst the Project Steering Committee met once in 2014, twice in 2015 and twice in 2016, there have been no meetings since August 2016, shortly after the Mid-term Review was finalised. This limited the effective input of relevant stakeholders and minimised capacity for M&E, inter-agency co-ordination and cross-sectoral dissemination of strategic findings.


Tag: Efficiency Human and Financial resources Monitoring and Evaluation Project and Programme management

21.

A Mid-term Review was undertaken 6 months later than planned because, as reported in PIR 2015, the National Project Co-ordinator was too busy with day-to-day operational activities to support the review.The MTR produced a set of recommendations and the management response was shared with the evaluation team. Many recommendations were simply to implement planned activities that had not progressed satisfactorily. The management response to one of these activities — weather-index insurance schemes have been implemented by private companies in collaboration NMA states that this has been achieved.


Tag: Efficiency Human and Financial resources Monitoring and Evaluation Operational Efficiency Project and Programme management

22.

UNDP and Implementing Partner implementation/execution

UNDP Execution is rated as: MODERATELY SATISFACTORYUNDP CO facilitated extensive international expertise to support equipment procurement and capacity building in support of project outcomes. The multi-country CIRDA programme, implemented by UNDP and funded by LDCF, provided support that would not be viable for a national project — without this support, this project in Ethiopia likely would not have been approved. CIRDA hired experts to collaborate on matters including hydromet technology, data management and collection, private sector engagement and communications.


Tag: Efficiency Human and Financial resources Procurement Project and Programme management Bilateral partners

23.

NMA Implementation is rated as: MODERATELY SATISFACTORY

The IP established eight task teams responsible for the implementation of project activities and the Responsible Parties each assigned a focal person with responsibility to deliver project outputs, while remaining accountable to the National Project Co-ordinator. However, as outlined in several previous sections, despite budgetary allocation from LDCF, allocation of offices at NMA and procurement of office equipment, no Project Management Unit (PMU) was established by the IP.


Tag: Effectiveness Operational Efficiency Oversight Project and Programme management Capacity Building

24.

0.1Project Results

0.1.1Achievement of objective and outcomes: Effectiveness

 Overall achievement of project objective and outcomes is rated as: SATISFACTORY

It was difficult to evaluate the overall attainment of the project objective and outcomes since objective level indicators were difficult to measure and some outcome level indicators were not effectively monitored. Whilst it was not possible to assign a quantified score, through a process of triangulation, a qualitative rating of satisfactory was reached. The evaluation of achievements is summarised below by objective and outcome level indicators. Full details are compiled in Annex 5.7.

Project Objective is rated as: SATISFACTORY

Objective: To strengthen the climate monitoring capabilities, early warning systems and available information for responding to climate shocks and planning adaptation to climate change in Ethiopia


Tag: Climate Change Adaptation Effectiveness Capacity Building

25.

Objective indicator 0.2: Domestic finance committed to the relevant institutions to monitor extreme weather and climate change

Target

Domestic target financing is 100M Birr per year

Rating

Moderately Satisfactory

Whilst a baseline domestic finance figure was identified in the Prodoc (66M Birr), PIRs did not report progress towards the target and the MTR did not identify this omission as a constraint to effective M&E, scoring progress towards this target as highly satisfactory without quantifiable evidence. The project team could not satisfactorily explain how the domestic finance target was set. However, NMAs sector specific GTP II (2015-2019) identifies a recurrent (O&M) budget projection (target) of 97.5 M Birr for the 2016/17 fiscal year and 126.5 M Birr for the 2017/18 fiscal year. In a personal comment from NMAs Director General, the recurrent budget for 2017/18 is 80M Birr. HWQDs GTP II identifies a total government budget of 7.6 M Birr for the 2017/18 fiscal year but does not break this down into capital expenditure and recurrent budget.  


Tag: Climate Change Adaptation Effectiveness National Institutions

26.

Outcome 1 is rated as: MODERATELY SATISFACTORY

Outcome 1: Percentage of national coverage of weather/climate and hydrological monitoring infrastructure

Outcome indicator 1.1: Percentage of national coverage of weather/climate and hydrological monitoring infrastructure

Target

Increase in 24% national coverage to take steps in achieving NMAs optimal monitoring arrangements as defined in feasibility studies

Rating

Moderately Satisfactory

The end of project target was erroneously articulated — it is not representative of the % increase in national coverage of hydromet monitoring infrastructure. It is simply the target % increase in the number of functioning manual meteorological stations (1240) against the baseline (1000) = 24%. Manual meteorological stations are not the focus of this project.

 

National coverage of AWS has increased by 57% however, at the time of reporting, four of these AWS were not transmitting data, reducing this increase in national coverage to 51%. There is Insufficient information to calculate increase in national coverage of hydrological stations with telemetry. Despite requesting information, HWQD did not report the status of telemetry stations with radar modems. Of the 40 GPRS equipped hydrological stations procured, only 35 were configured and 32 were installed, representing an increase in national coverage of 700%. However, of these 32, only 22 are transmitting data, reducing this increase in national coverage to 450%.

 


Tag: Climate Change Adaptation Effectiveness Monitoring and Evaluation

27.

Outcome indicator 1.2 Frequency and timeliness of climate related data availability

Target

Daily

Rating

Highly Satisfactory

AWS and hydrological stations have the capacity to transmit data to the Federal server every 15 minutes. However, data is transmitted only when the GPRS network is available. Although out of the control of the project, on several occasions, and in many localities, access to Ethio Telecom’s mobile data network was blocked by the government, often for many months at a time.


Tag: Climate Change Adaptation Effectiveness Monitoring and Evaluation

28.

Outcome 2: Efficient and effective use of hydro-meteorological and environmental information for early warnings and long-term development plans

Outcome indicator 2.1: population with access to improved climate information and flood and drought warnings (disaggregated by gender)

Target

Percentage increase in population who have access to improved EWS/CI (50% increase for women, and 50% increase for men)

Rating

Moderately Satisfactory

The project did not report effectively against this indicator. The means of measurement identified in the Prodoc was a gender disaggregated beneficiary survey, including vulnerability reduction assessment relative to the baseline. These surveys and assessments were not undertaken.

However, the significant increase in hydromet observation infrastructure, coupled with targeted capacity building activities for NMA, HWQD and NDRMC — including numerous training events, workshops and international study tours — has improved the accuracy and availability of climate information throughout the country. Awareness creation workshops were organised at all meteorological branch directorates for regional government policy makers, women and local community representatives on how to interpret climate information and make use of it to minimise climate related risks. Climate forecasts and impact advisories are disaggregated at the regional and zonal level and disseminated three times a week to a wide variety of stakeholders. These forecasts are used at the federal and regional levels to advise national and regional government offices in decision-making.

This was evident during El Niño in 2015/16 when project resources supported the establishment of a national and regional El Niño monitoring and forecasting task force. During this time, accurate forecasting and monitoring supported decision makers and NDRMC to minimise the impact of flooding.

PIR 2017 states that, according to the Ministry of Agriculture, climate information flows to the lowest government structure (kabele’s). It further states that during the El Niño, 75% of farmers, agricultural extension agents and local disaster risk managers received hydromet advisories and forecasts every two days. However, it is not possible to corroborate this statement since a survey of end-users has not been undertaken.

 


Tag: Climate Change Adaptation Disaster Risk Reduction Natural Disaster Country Government

29.

Outcome indicator 2.2: Development frameworks (e.g. GTP) that integrate climate information in the formulation

Target

At least 2 of the PRSP policy briefs incorporate analyses of risk maps and/or climate change projections influencing long-term planning proposals

Rating

Satisfactory

The objective and outcome targets of the project have been incorporated in the pillars of NMA and HWQD sector-specific GTPs. NMAs GTP II is aligned with global and regional perspectives including the Integrated African Strategy on Meteorology (IASM). HWQDs GTP II aims to upgrade all hydrology stations with telemetry. Ethiopia’s second National GTP (2015-2020) incorporates targets for delivery of meteorological forecasting and early warning services including: “preparation and dissemination of short duration weather forecasting reports twice a day; midterm weather forecast on daily basis; 1-5 days cities weather forecast which could be updated daily as well as regional midterm weather forecast which could be updated yearly”.


Tag: Climate Change Adaptation Disaster Risk Reduction Natural Disaster Effectiveness Risk Management

30.

Outcome indicator 2.3: Sector-specific EW products and strategies that integrate climate risks

Target

Development of at least one tailored climate product and presentation of market research plan on how to implement mobile phone based agricultural advisories, both supporting targeted weather/climate service delivery

Rating

Unsatisfactory

Progress towards this indicator was not effectively monitored or documented and the project team had very little knowledge of progress and achievements.  NMAs 2016 half-year progress report states that dialogue was underway between NMA and Ethio Telecom, the provider of internet, SMS and other form of telecommunications in Ethiopia. These discussions did not progress.  Mobile communications are state run in Ethiopia and the project was unable to develop a market-based rationale for Ethio Telecom’s support. It is unclear how much resource was dedicated to these discussions — NMAs Meteorological Research and Studies Directorate stated that no activities mobile phone-based advisories activities were implemented under this project. The directorate also confirmed that activities were being taken forward under an ADB-Climdev funded project and that a technology and needs assessment, contracted to an Indian company, was ongoing.

The project aimed to approach the private sector to increase existing fee structures with existing customers and make additional funds available from insurance companies who expressed a readiness to pay in the national consultation (during the project preparation phase) for weather forecasting services and products. These sectors included includes civil aviation, hydropower, tourism and farmers’ associations. The project supported a public-private partnerships finance forum in Adama however, no further details were available, and NMA is currently undertaking a study on payment for meteorological services. One public-private partnership is under development. However, the modality of this partnership with the aviation sector, moreover with Ethiopian Airlines, has not been accepted by contracting parties as the ongoing payment for services was found to be disproportionately small. NMA has initiated negotiations and plans to charge 35M Birr/year for its services.

The project planned to develop public-private partnerships with insurance companies with a special focus on weather indexing methods. The updated MTR management response (dated May 2018) stated that “New weather-index insurance schemes implemented by private companies in collaboration with NMA”. However, this there is no mention of this activity in any of the PIRs or NMA quarterly reports and the project team could not provide any further information. The comment in the MTR management response may have been in relation to a 2016 workshop to evaluate NMA projects and activities funded by international development organisations. One of the projects presented at this workshop was a GeoData for Agriculture and Water funded project: Weather index-based Credit Insurance pilot project in Amhara, Oromia, SNNP and Tigray regions. Amongst the project’s objectives was “production of Geodata for Innovative Agricultural Credit Insurance Scheme (GIACIS) to Kifiya for premium collection and claim payout according to the respective Kebeles’ drought severity identified on the satellite data of Normalized Difference Vegetation Index (NDVI)”.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Effectiveness Partnership Strategic Positioning Private Sector

31.

Outcome 3: Effective and efficient project management and knowledge management, communication, stakeholder engagement

Outcome indicator 3.1: Presence of effective and integrated project management unit

Target

Put in place an integrated project management unit, stakeholder platform, communication and knowledge management system 

Rating

Unsatisfactory

Despite budgetary allocation from LDCF, allocation of offices at NMA and procurement of office equipment, no PMU was established by the IP. Therefore, the project lacked a dedicated Project Manager, Administrative and Financial Assistant, and Monitoring & Evaluation Expert. This had a significant impact on project implementation, and monitoring & evaluation.

 

A Project Steering Committee was established and met once in 2014, twice in 2015 and twice in 2016. However, there were no further meetings after August 2016, shortly after the Mid-term Review was finalised. The Project Steering Committee did not effectively perform its function — meetings were not well attended by appointed stakeholders; several members never attended, two members interviewed by the evaluation team could not recollect what the project was about and one stated they were unaware of their responsibilities since no ToR was circulated. However, project advisory committees were established in NMA, NDRMC and HWQD to convene technical consultations.

 

A communication and knowledge management system was not established. NMA launched a bid process but did not receive any tenders. Subsequently, NMA reported that they transferred this budget to UNDP and that UNDP identified a winner. While it is reported that an individual consultant commenced the work, the evaluation team has not seen any outputs.


Tag: Effectiveness Efficiency Communication Knowledge management Project and Programme management

32.

Relevance is rated as: RELEVANT

The project has made significant contributions to Ethiopia’s NAPA (2007), is highly relevant to national development goals and supports the implementation of key policies.

The project has directly supported implementation of NAPA priority 2: “Strengthening/enhancing drought and early warning systems in Ethiopia” and NAPA priority 6: “Capacity building program for climate change adaptation”. Consistent with Ethiopia’s Growth and Transformation Plan (GTP 2016-2020), the project has made important contributions to the implementation of the food security disaster prevention and preparedness strategy, by making early warning systems more accessible and up-to-date, and has supported targets for delivery of meteorological forecasting and early warning services including: “preparation and dissemination of short duration weather forecasting reports twice a day; midterm weather forecast on daily basis; 1-5 days cities weather forecast which could be updated daily as well as regional midterm weather forecast which could be updated yearly”.

The objective and outcome targets have been incorporated in the pillars of NMA and HWQD sector-specific GTPs (II) and NMAs GTP II is aligned with global and regional perspectives, including the Integrated African Strategy on Meteorology (IASM).

The project is highly relevant to the GoE National Policy and Strategy on Disaster Risk Management (2013) and has contributed to two of the three pillars of disaster risk management — prevention and preparedness. It is also highly relevant to Ethiopia’s Climate Resilient Green Economy Strategy (2011), which recognises that building a climate resilient economy requires effective adaptation to minimise potential damage and maximise potential benefits.


Tag: Climate Change Adaptation Natural Disaster Relevance Country Government Capacity Building

33.

Efficiency

 Efficiency is rated as: MODERATELY SATISFACTORYInvestments in hydromet observational equipment and institutional capacity building are considered cost-effective. Through the LDCF supported multi-country CIRDA programme, UNDP CO facilitated extensive international expertise to support equipment procurement and capacity building in support of project outcomes. Long-Term Agreements, negotiated by CIRDA with suppliers of hydromet observational equipment and services, streamlined UNDP CO procurement, reducing costs and delivery lead times. And with CIRDA support, UNDP facilitated extensive international expertise to support cost-effective institutional capacity building through study-tours and on-site training. While no co-finance was secured, the project shares complimentary outcomes with several parallel projects and the GoE has leveraged additional finances from international development agencies during project implementation. These additional finances are programmed to further advance implementation of NMAs second sectoral GTP (2015-2019).  


Tag: Efficiency Partnership Procurement Capacity Building Institutional Strengthening

34.

Country ownership

The project was developed in alignment with national strategy. Ethiopia’s first GTP (2010-2015) acknowledged the impact of climate change on the country’s economy and ability to achieve Millennium Development Goal (MDG) targets. Ethiopia’s NAPA identified “Strengthening/enhancing drought and early warning systems in Ethiopia” as its second priority; and “Capacity building program for climate change adaptation” as its sixth. The CRGE (2011) —which superseded the Carbon Neutral Climate Resilient Economy (CNCRE, 2010), which itself was developed as part of the NAPA implementation process — identifies the importance of building climate change resilience to achieve the country’s economic growth targets and is based on the National Appropriate Mitigation Actions (NAMA) and Ethiopia’s Programme of Adaptation to Climate Change (EPACC). The EPACC builds on the NAPA and to deliver best practice actions for climate change adaptation. 


Tag: Sustainability Communication Human and Financial resources Ownership Strategic Positioning Capacity Building

35.

3.3.5     Mainstreaming

The project objective is directly aligned with Pillar II of Ethiopia’s latest UN Development Assistance Framework (UNDAF, 2016-2020): Resilience & Green Economy; specifically, “better prepare, respond to and recover from emergencies and disasters”. It is also aligned directly with Pillar II of UNDPs Country Programme Document (CPD, 2016-2020): Climate change and resilience-building and contributes to Pillar III: Strengthening democratic governance and capacity development, through its activities focused on institutional capacity building. The project, and UNDAF, are fully aligned with the national development priorities articulated in the second Growth and Transformation Plan (GTP II, 2016-2020).


Tag: Climate Change Adaptation Disaster risk management Resilience building Effectiveness Local Governance Country Government UN Agencies Institutional Strengthening

36.

Sustainability

Overall Sustainability is rated as: MODERATELY LIKELY

Sustainability is considered here as the likelihood of continued benefit after the project ends. This section considers four risk dimensions: financial, socio-economic, institutional and governance, and environmental. Since all are critical, the overall rating given should not be higher than the lowest rated dimension. All dimensions were rated as moderately likely.


Tag: Efficiency Sustainability Human and Financial resources Operational Efficiency Project and Programme management

37.

Socio-economic sustainability is rated as: MODERATELY LIKELY

A significant number of observers/technicians — responsible for taking manual measurements at hydromet stations — interviewed by the evaluation team stated that there is little community-level awareness of the long-term value of hydromet observational equipment. Communities are generally unaware of the impact equipment has on improving climate information and early warning advisories so are generally unaware how the equipment may benefit them.


Tag: Climate Change Adaptation Effectiveness Sustainability Awareness raising

38.

Institutional Framework and governance sustainability is rated as: MODERATELY LIKELY

Project outcomes are institutionalised into NMA, the Ministry of Water, irrigation and Electricity and the Ministry of Agriculture, and recurrent budgets are allocated to NMA, HWQD and NDRMC for climate information and early warning services. Ethiopia’s second National GTP (2015-2020) incorporates targets for delivery of meteorological forecasting and early warning services and outcome targets are incorporated in the pillars of NMA, HWQD and NDRMC sector-specific GTPs.ta cannot be transmitted.


Tag: Climate Change Adaptation Natural Disaster Effectiveness Sustainability Country Government

39.

Environmental sustainability is rated as: MODERATELY LIKELY

Climate shocks threatened implementation of activities.  NMA noted that the 2015/16 El Niño diverted the resources of all project partners for several months, as operational capacity was diverted to monitoring extreme climate events and issuing rainfall and flooding advisories to government stakeholders and civil society.

Generally, the hydrometeorological infrastructure installed is resilient to Ethiopia’s environmental conditions. However, a number of hydrological gauge stations utilise shaft encoder technology. During heavy rainfall or flood events, sedimentation build up stops these gauge stations from taking accurate measurements. Regular maintenance is required during the rains to ensure reliability.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Sustainability

40.

Impact

Impact is rated as: MINIMALImpact analysis requires the availability of process indicators and/or verifiable data on changes in socioeconomic or environmental state. These indicators and/or verifiable data were not developed/collected by the project and the GEF Focal-area tracking tool, which is designed to greatly aid in the assessment of impact, was not utilised by UNDP. Therefore, while the rating assigned is based on the findings of the evaluation, it is subjective.


Tag: Climate Change Adaptation Disaster risk management Natural Disaster Impact Country Government Capacity Building

41.

NMA also confirmed budgetary limitations for O&M. For the 2017/18 fiscal year, NMAs total budget was approximately 175M Birr, of which 95M was allocated to capital projects and 80M was allocated to recurrent costs such as salaries and O&M. However, NMAs sector specific GTP (2015-2019) identifies a 2017/18 budget projection (target) of 278M, of which 126M is allocated to recurrent costs, rising to a total budget target of 318M in 2019/20, of which 178M is allocated to recurrent costs. These sharp increases in O&M targets reflect NMAs GTP II target of 700 operational AWS by 2019. At the time of this report, approximately 250 are operational.

The project had an objective level target so secure 100M Birr of domestic finance for O&M activities. This target fell below NMAs 2017/18 fiscal year target of 126M Birr. Budgetary shortfalls for O&M present a significant financial risk to sustainability. HWQDs GTP II identifies a total government budget of 7.6 M Birr for the 2017/18 fiscal year but does not break this down into capital expenditure and recurrent budget. As stated under indicator 2.3, the project planned to develop public-private partnerships to provide on-going revenue for weather forecasting services and products, including weather-indexing for insurance companies. As reported under section 3.3.1, little progress was made towards this indicator.  While NMA is currently undertaking a study on payment for meteorological services, during project implementation only one public-private partnership was under development, with Ethiopian Airlines. However, the modality of this partnership is yet to be agreed.

In the short term, some O&M costs will be covered by various ongoing and planned international development aid funded projects. However, this is not sustainable in the medium to long-term.


Tag: Effectiveness Sustainability Human and Financial resources Project and Programme management

Recommendations
1

Project design

Identify actions to safeguard public goods

A risk analysis was undertaken during the project preparation phase and is presented in Annex I of the Prodoc. However, while NMA and HWQD emphasised a critical shortfall in O&M budgets, this sustainability risk was not included in the analysis or adequately considered in project design. Significant resources were invested in procurement and installation of hydromet observation equipment, but risk of equipment theft, vandalism and lack of routine maintenance was not addressed. While it is recognised that, once equipment is handed over to the government it becomes their responsibility, the project should have investigated and budgeted specific actions to minimise risk in the short-term, while longer-term sustainability is developed.

 

2

Evaluation Recommendation or Issue 2: Actions to follow up or reinforce initial benefits from the project:

3

Accurately identify baselines and measurable indicators

The methodology to quantify the baseline capacity assessment score, and measure progress toward the target, was not identified. While scores were assigned, they were had no meaning in the absence of a methodology. Accordingly, it was not possible to quantify progress towards the project’s objective.

4

Include outputs and output level indicators in the PRF

The absence of output level indicators made it difficult to effectively monitor implementation of activities and identify adaptive management in a timely manner.

5

Project implementation, monitoring & evaluation

Establish an effective Project Steering Committee

The PSC was not well attended by its appointed stakeholders and did not adequately fulfil its function. Several members never attended, two members interviewed by the evaluation team could not recollect the project and one stated they were unaware of their responsibilities, since no ToR was circulated. The PSC has not met since August 2016. The roles and responsibilities of PSC members should be clearly communicated to maximise participation and buy-in of stakeholders.

6

Establish an Effective Project Management Unit

Despite budgetary allocation from LDCF, allocation of offices at NMA and procurement of office equipment, no PMU was established by the IP. Therefore, the project lacked a dedicated project manager, administrative and financial assistant, and monitoring & evaluation expert. The part-time national project co-ordinator was too busy with day-to-day activities to adequately provide project implementation oversite, and the capacity of focal points at HWQD and NDRMC was also stretched.

7

Make use of the GEF tracking tool

The GEF Focal Tracking Tool was not completed at any stage of the project. It is an important M&E tool for supporting measurement of progress towards project outcomes and outputs, and for identifying requirements for adaptive management. It also supports the development of impact level indicators. Its omission from this project contributed to evaluators not being able to assign an impact rating.

8

Identify and assess risks at hydromet sites

During the field mission, the evaluators learned of several incidences of theft and/or vandalism of critical hardware, including solar panels and GPRS units. Of the 32 hydrological stations with GPRS telemetry installed, only 22 were functional by the end of the project. Solar panels and/or GPRS equipment had been stolen at a minimum of four of these stations. Of the 40 AWS installed, four were not functional at the end of the project and solar panels had been stolen at a minimum of two. While the causes of malfunction of the remaining AWS is not known to the evaluators, a further two hydrological sites malfunctioned due to poor internet connectivity, and one malfunctioned due to sedimentation build up in the gauge. Multi-hazard risk assessments should be undertaken to identify broad and site-specific mitigation actions.

9

Undertake community-based awareness raising of the value of public goods

Generally, there is insufficient community-level awareness of the purpose and value of hydromet observational equipment. Awareness raising campaigns should be implemented to sensitise communities to the role these public goods play in improving the accuracy of climate information and early warnings, and how this benefits civil society. By raising awareness, the incidence of theft and vandalism should be reduced. Several observers suggested an effective medium for awareness raising would be to communicate through religious and other community leaders. Inauguration ceremonies for local stakeholders have taken place at some AWS sites.  Apparently, these have been effective at raising local awareness. Opportunities for school visits to AWS and hydrological sites, and the erection of interpretative signage should also be explored.

10

Where feasible, employ guard at sites situated away from settlements

For sites situated away from settlements and exposed to high risk of theft/vandalism, consideration should be given to employing guards. The minimal cost would be offset by a reduction in capital investments to replace stolen/vandalised equipment. This cost should be built in to recurrent budgets.

11

Maximise land-use at AWS sites

AWS sites are situated on 1 ha land holdings and are enclosed by fencing. The majority of each plot is unused. The sites could host meteorological and climate change education programmes as an effective means of community outreach. During the field mission, the team visited Dangila AWS in Amhara. The observer, using her own resources, had established a small environmental education facility. It was extremely popular with the local community and hosted regular groups, including from the local school. There was evidence of significant impact on local awareness raising. Furthermore, by fully utilising land at AWS sites, NMA would have further leverage to push back against land claims, which have significantly increased in recent years due to escalating land prices.

12

Develop a sustainable finance strategy

While the project aimed to approach the private-sector to increase fee structures with existing customers and make additional funds available from new customers, little progress was made. NMA is currently undertaking a study on payment for meteorological services. Through consultation with parallel projects, and building on global best practice, this study should be developed to identify the feasibility of implementing private-sector financial mechanisms. If private-sector financial mechanisms are found to be feasible, a sustainable finance strategy, based on these findings, should be developed and implemented.

13

Develop a strategy to implement mobile phone-based agricultural advisories

The project aimed to develop a market research plan to implement mobile phone-based agricultural advisories. However, few discussions were held with Ethio Telecom — Ethiopia’s mobile telecommunications provider. A parallel ADB-Climdev funded project is undertaking a technology and needs assessment. This assessment should be developed into a strategy to identify the feasibility of implementing mobile phone-based agricultural advisories.

14

Decentralise operations and maintenance to the regional level

The installation and maintenance of hydromet observational equipment is centralised at the Federal level. There is significant evidence that this is inefficient and ineffective. Capacity for O&M is weak at the regional level and federal level technicians have limited budgets to travel. Equipment failures or theft, though reported, are regularly overlooked, weakening national and regional capacity to generate climate information and early warnings. By decentralising O&M, regional authorities would be empowered as stewards of hydromet observational infrastructure and O&M costs would be minimised. Senior staff at NMA and HWQD recognise that, as Ethiopia’s climate information infrastructure continues to expand, decentralisation is a necessity. As a first step, a capacity needs assessment should be undertaken at all regional NMA and HWQD offices.

15

Develop an awareness raising communications strategy

Establishing a network of hydromet observational equipment and building capacity for forecasting and early warnings are significant achievements of this project. However, influencing behavioural change to effectively use information and make informed decisions is another challenge. A communications strategy should be developed to identify opportunities for awareness raising platforms at regional, zonal and woreda levels. Attention should be paid to opportunities for engaging with schools, since building awareness in children is an effective way to communicate with adults, particularly in rural communities.

16

Ensure hydrological observational equipment is fit for purpose

A number of hydrological gauge stations use shaft encoder technology which is susceptible to sedimentation build-up. Sedimentation stops these gauge stations from recording accurate measurements and regular maintenance is required during rains to ensure reliability. Since O&M capacity in Ethiopia is limited, these gauge stations may remain inactive for several months, reducing capacity to monitor river levels and issue flood advisories. Site-specific environmental conditions should be carefully considered when selecting equipment requirements.

1. Recommendation:

Project design

Identify actions to safeguard public goods

A risk analysis was undertaken during the project preparation phase and is presented in Annex I of the Prodoc. However, while NMA and HWQD emphasised a critical shortfall in O&M budgets, this sustainability risk was not included in the analysis or adequately considered in project design. Significant resources were invested in procurement and installation of hydromet observation equipment, but risk of equipment theft, vandalism and lack of routine maintenance was not addressed. While it is recognised that, once equipment is handed over to the government it becomes their responsibility, the project should have investigated and budgeted specific actions to minimise risk in the short-term, while longer-term sustainability is developed.

 

Management Response: [Added: 2018/11/28] [Last Updated: 2021/01/13]

partially agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Identify actions to safeguard public goods;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP, NMA,and NDRMC 2019/12 Completed Completed History
Accurately identify baselines and measurable indicators;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP & Government 2020/12 Completed Completed History
Include outputs and output level indicators in the PRF;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP and Government 2020/12 Completed Completed History
Establish an Effective Project Management Unit;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP and NMA 2020/12 Completed Completed History
Make use of the GEF tracking tool;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP 2020/08 Completed Completed History
2. Recommendation:

Evaluation Recommendation or Issue 2: Actions to follow up or reinforce initial benefits from the project:

Management Response: [Added: 2018/11/28] [Last Updated: 2018/12/21]

Agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Identify and assess risks at hydro met sites;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP, HWQD and NDMC 2019/06 Completed Completed History
Undertake community-based awareness raising of the value of public goods;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP , NMA , HWQD of the MoWIE 2019/12 Completed Completed History
Undertake community-based awareness raising of the value of public goods;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP, NMA , HWQD, of the MoWIE 2020/12 Completed Completed History
Maximise land-use at AWS sites;
[Added: 2018/11/28] [Last Updated: 2020/08/03]
UNDP, NMA and CEFCC 2020/12 Completed Completed History
Where feasible, employ guard at sites situated away from settlements
[Added: 2018/11/28] [Last Updated: 2020/08/03]
NMA/HWQD and NDRMC 2019/06 Completed Completed History
3. Recommendation:

Accurately identify baselines and measurable indicators

The methodology to quantify the baseline capacity assessment score, and measure progress toward the target, was not identified. While scores were assigned, they were had no meaning in the absence of a methodology. Accordingly, it was not possible to quantify progress towards the project’s objective.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

4. Recommendation:

Include outputs and output level indicators in the PRF

The absence of output level indicators made it difficult to effectively monitor implementation of activities and identify adaptive management in a timely manner.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

5. Recommendation:

Project implementation, monitoring & evaluation

Establish an effective Project Steering Committee

The PSC was not well attended by its appointed stakeholders and did not adequately fulfil its function. Several members never attended, two members interviewed by the evaluation team could not recollect the project and one stated they were unaware of their responsibilities, since no ToR was circulated. The PSC has not met since August 2016. The roles and responsibilities of PSC members should be clearly communicated to maximise participation and buy-in of stakeholders.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

6. Recommendation:

Establish an Effective Project Management Unit

Despite budgetary allocation from LDCF, allocation of offices at NMA and procurement of office equipment, no PMU was established by the IP. Therefore, the project lacked a dedicated project manager, administrative and financial assistant, and monitoring & evaluation expert. The part-time national project co-ordinator was too busy with day-to-day activities to adequately provide project implementation oversite, and the capacity of focal points at HWQD and NDRMC was also stretched.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

7. Recommendation:

Make use of the GEF tracking tool

The GEF Focal Tracking Tool was not completed at any stage of the project. It is an important M&E tool for supporting measurement of progress towards project outcomes and outputs, and for identifying requirements for adaptive management. It also supports the development of impact level indicators. Its omission from this project contributed to evaluators not being able to assign an impact rating.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

8. Recommendation:

Identify and assess risks at hydromet sites

During the field mission, the evaluators learned of several incidences of theft and/or vandalism of critical hardware, including solar panels and GPRS units. Of the 32 hydrological stations with GPRS telemetry installed, only 22 were functional by the end of the project. Solar panels and/or GPRS equipment had been stolen at a minimum of four of these stations. Of the 40 AWS installed, four were not functional at the end of the project and solar panels had been stolen at a minimum of two. While the causes of malfunction of the remaining AWS is not known to the evaluators, a further two hydrological sites malfunctioned due to poor internet connectivity, and one malfunctioned due to sedimentation build up in the gauge. Multi-hazard risk assessments should be undertaken to identify broad and site-specific mitigation actions.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

9. Recommendation:

Undertake community-based awareness raising of the value of public goods

Generally, there is insufficient community-level awareness of the purpose and value of hydromet observational equipment. Awareness raising campaigns should be implemented to sensitise communities to the role these public goods play in improving the accuracy of climate information and early warnings, and how this benefits civil society. By raising awareness, the incidence of theft and vandalism should be reduced. Several observers suggested an effective medium for awareness raising would be to communicate through religious and other community leaders. Inauguration ceremonies for local stakeholders have taken place at some AWS sites.  Apparently, these have been effective at raising local awareness. Opportunities for school visits to AWS and hydrological sites, and the erection of interpretative signage should also be explored.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

10. Recommendation:

Where feasible, employ guard at sites situated away from settlements

For sites situated away from settlements and exposed to high risk of theft/vandalism, consideration should be given to employing guards. The minimal cost would be offset by a reduction in capital investments to replace stolen/vandalised equipment. This cost should be built in to recurrent budgets.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

11. Recommendation:

Maximise land-use at AWS sites

AWS sites are situated on 1 ha land holdings and are enclosed by fencing. The majority of each plot is unused. The sites could host meteorological and climate change education programmes as an effective means of community outreach. During the field mission, the team visited Dangila AWS in Amhara. The observer, using her own resources, had established a small environmental education facility. It was extremely popular with the local community and hosted regular groups, including from the local school. There was evidence of significant impact on local awareness raising. Furthermore, by fully utilising land at AWS sites, NMA would have further leverage to push back against land claims, which have significantly increased in recent years due to escalating land prices.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

12. Recommendation:

Develop a sustainable finance strategy

While the project aimed to approach the private-sector to increase fee structures with existing customers and make additional funds available from new customers, little progress was made. NMA is currently undertaking a study on payment for meteorological services. Through consultation with parallel projects, and building on global best practice, this study should be developed to identify the feasibility of implementing private-sector financial mechanisms. If private-sector financial mechanisms are found to be feasible, a sustainable finance strategy, based on these findings, should be developed and implemented.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

13. Recommendation:

Develop a strategy to implement mobile phone-based agricultural advisories

The project aimed to develop a market research plan to implement mobile phone-based agricultural advisories. However, few discussions were held with Ethio Telecom — Ethiopia’s mobile telecommunications provider. A parallel ADB-Climdev funded project is undertaking a technology and needs assessment. This assessment should be developed into a strategy to identify the feasibility of implementing mobile phone-based agricultural advisories.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

14. Recommendation:

Decentralise operations and maintenance to the regional level

The installation and maintenance of hydromet observational equipment is centralised at the Federal level. There is significant evidence that this is inefficient and ineffective. Capacity for O&M is weak at the regional level and federal level technicians have limited budgets to travel. Equipment failures or theft, though reported, are regularly overlooked, weakening national and regional capacity to generate climate information and early warnings. By decentralising O&M, regional authorities would be empowered as stewards of hydromet observational infrastructure and O&M costs would be minimised. Senior staff at NMA and HWQD recognise that, as Ethiopia’s climate information infrastructure continues to expand, decentralisation is a necessity. As a first step, a capacity needs assessment should be undertaken at all regional NMA and HWQD offices.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

15. Recommendation:

Develop an awareness raising communications strategy

Establishing a network of hydromet observational equipment and building capacity for forecasting and early warnings are significant achievements of this project. However, influencing behavioural change to effectively use information and make informed decisions is another challenge. A communications strategy should be developed to identify opportunities for awareness raising platforms at regional, zonal and woreda levels. Attention should be paid to opportunities for engaging with schools, since building awareness in children is an effective way to communicate with adults, particularly in rural communities.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

16. Recommendation:

Ensure hydrological observational equipment is fit for purpose

A number of hydrological gauge stations use shaft encoder technology which is susceptible to sedimentation build-up. Sedimentation stops these gauge stations from recording accurate measurements and regular maintenance is required during rains to ensure reliability. Since O&M capacity in Ethiopia is limited, these gauge stations may remain inactive for several months, reducing capacity to monitor river levels and issue flood advisories. Site-specific environmental conditions should be carefully considered when selecting equipment requirements.

Management Response: [Added: 2021/01/08] [Last Updated: 2021/01/13]

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org