Terminal Evaluation for Early Warning System

Report Cover Image
Evaluation Plan:
2016-2020, Uganda
Evaluation Type:
Final Project
Planned End Date:
06/2018
Completion Date:
08/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
30,000

Share

Title Terminal Evaluation for Early Warning System
Atlas Project Number: 00076999
Evaluation Plan: 2016-2020, Uganda
Evaluation Type: Final Project
Status: Completed
Completion Date: 08/2018
Planned End Date: 06/2018
Management Response: Yes
Focus Area:
  • 1. Environment & Sustainable Development
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 5.4. Preparedness systems in place to effectively address the consequences of and response to natural hazards (e.g. geo-physical and climate related) and man-made crisis at all levels of government and community
SDG Goal
  • Goal 13. Take urgent action to combat climate change and its impacts
SDG Target
  • 13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries
  • 13.2 Integrate climate change measures into national policies, strategies and planning
  • 13.3 Improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning
Evaluation Budget(US $): 30,000
Source of Funding:
Evaluation Expenditure(US $): 30,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Richard Sobey International Consultant-Team Leader timosobey@gmail.com
Michael Mbogga National Consultant michael.mbogga@gmail.com UGANDA
Richard Sobey M&E International Consultant timosobey@gmail.com
Michael Mbogga M&E national Consultant michael.mbogga@gmail.com UGANDA
GEF Evaluation: Yes
GEF Project Title: Strengthen climate information and early warning systems in Africa for climate resilient development and adaptation to climate change รข?? Uganda
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: FSP
GEF Phase: GEF-5
GEF Project ID: 4993
PIMS Number: 5094
Key Stakeholders: Uganda National Meteorological Authority
Countries: UGANDA
Comments:

See minute 2.3 SCIEWS presentation as per attached joint board minutes of 13-15th December 2016.

Lessons
1.

UNMA’s modernisation under SCIEWS has been successful.  SCIEWS has highlighted the future minimum required status for CI and EWSs provision, and has successfully been testing the systems for the last 18 months.  The capability in forecasting and its reliability has significantly improved (due to equipment upgrade and practice in forecasting).  Twenty new AWSs are transmitting data to UNMA.  Twelve synoptic stations are operational.  These are ISO accredited to WMO standards and are contributing to East Africa / international weather forecasting.

 


2.

UNMA’s CD gathering network was not significantly expanded, but AWSs were placed where needed and automated under SCIEWS, which was one of the main aims.  An automatic message switching system (AMSS) was installed to share synoptic CD globally in real time.  Before SCIEWS, the data from only one station was transmitted manually on a global level, with no data returned.  Now ten (out of 12) AWSs transmit automatically under the global AMSS with all global data returned as required.

 


3.

Based on their six-monthly portfolio review, MoFPED considered SCIEWS as a flagship or benchmark project in terms of delivery.  UNMA as the lead implementing partner has met stakeholder expectations.  However, the challenge remains for planning and financial commitment from government.  O&M costs are an on-going issue (including re-calibration, spares / batteries & periodic maintenance).

 


4.

The primary source of climate data (CD) and climate information (CI) is UNMA.  It is then repackaged by NECOC and MAAIF.  Districts use consolidated NECOC bulletins for EW and disaster risk management.  The confidence in CI / EWS services has been strengthened under SCIEWS.  However, for CI dissemination from district to sub-county to parish to farmer, there is a variable picture, with insufficient evidence to show significantly improved access to CI at the local level. 

 


5.

The lack of an up to date telecoms agreement for the NMC is a concern.  The lack of bandwidth and uninterrupted connectivity means that key systems for short-term weather forecasting (for use by aviation at Entebbe and across Uganda) are not fully operational.

 


6.

DWRM was very late in installing the AWLSs with some not yet transmitting data.  This is due to the telemetry units not being installed in some stations due to a known high risk of theft and vandalism, or not operational due to a limited power supply.  Added to this, DWRM missed the 2018 government planning window which should have included the request for an O&M budget for these new SCIEWS stations.  Thus, in spite of the efforts of SCIEWS, Uganda remains without an effective flood warning system.

 


7.

Mainstreaming of climate information remains an issue, mainly in terms of district-level development plans, within which climate adaptation measures are beginning to be described, but not the use of CI.  Catchment-based water resource management planning is at an early stage and not linked to NECOC district hazard vulnerability mapping, nor available CD / CI.  This needs facilitation.

 


Findings
1.

Detailed ratings are tabulated below in Exhibit 3.  

Exhibit 3: TE Ratings and Achievement Summary Table

Project:  Strengthening Climate Information & Early warning systems in Uganda

(GEF Project ID: 4993; UNDP PIMS ID: 5094)

TE Rating

Achievement Description

 

Outcomes/ Results

Results Overall Project Objective Achievement

Satisfactory

Objective: To strengthen the weather, climate & hydrological monitoring capabilities, EWSs and information for responding to extreme weather and planning adaptation to climate change

The overall TE Rating is Satisfactory

Justification:  The attainment of the project objective was primarily due to the achievement of Outcome 1 and 2 in terms of the provision of generation of climate data (CD) and in the generation of climate information (CI) and early warning information (EWI) through alerts and advisories.

UNMA has clearly built capacity in the provision of climate services, although it remains understaffed, in both technicians to maintain the new generation of weather stations, but also in climate analysts to prepare the forecasts.  These forecasts vary in frequency (15 minute, hourly, 6-hourly, daily, 3-day, 10-day, monthly, seasonal), locale (city, regional) and type (rainfall with above or below rainfall areas, and with advisories – for agriculture and water & energy in particular).  However, the present capacity to maintain the NMC in good working order appears somewhat limited, although the reasons are unclear.

UNMA’s budget has increased from US$2.5m in 2014/15 to ~US$9.0m in 2018/19, although a third of this is due to donor contributions.  UNMA now as a semi-autonomous authority has the mandate to raise revenue from the sale of its services, but the overriding reason for its existence is to provide climate information as a public service, which means that the two sides of the business should separate.

The indication is that whilst the newly formed UNMA is being heavily invested in, comparatively DWRM is struggling.  At present DWRM has AWLSs that it can’t bring to full operation for automatic data transmission (for flood warning) and that it can’t maintain due to lack of budget.  It is assumed that the newly planned flood modelling unit will re-vamp its fortunes.  Also, for DWRM, there appears to be a missing link between catchment-based planning (DWRM) and hazard mapping (NECOC).

There were two indicators individually rated:

                       - Capacity Scorecard – Satisfactory

                       - Finance committed to monitor climate - Satisfactory

Effectiveness - Outcome 1 Achievement

Satisfactory

Outcome 1:  Enhanced capacity of UNMA & DWRM to monitor & forecast extreme weather, hydrology and climate change 

The overall TE rating is Satisfactory

Justification:

% coverage of climate/weather and hydrological infrastructure – Satisfactory - From a baseline of 10%, UNMA’s national coverage of AWSs is now at 29%, with one station able to capture CD representative of at least two districts.  SCIEWS has contributed 63% of AWSs in the country.  Twenty AWSs were installed, with 29 other stations upgraded.  With the addition of an Automatic Message Switching System (AMSS), 12 synoptic stations are now able to transmit internationally on a quarter hour basis.

From a baseline of 4%, DWRM’s coverage of AWLS is now 40%.  Sixteen AWLSs were installed with 28 stations’ housing constructed for telemetry.  However, at some locations data is still collected manually due to the challenge of a regular power supply.

Frequency & timeliness of climate-related data – Satisfactory - For the transmission of climate data (CD), the AWSs transmit at near real time, with the synoptic stations transmitting every 15 minutes.  At NMC, the short-term weather forecasting is every 6 hours and as the weather changes.

DWRM’s AWLSs have data loggers, but at present telemetry is not installed, thus data collection is manual on a weekly or monthly basis.

For the analyzed weather and climate data, i.e. climate information (CI), the frequency of dissemination is every 10 days, monthly, seasonally with a two-week lead in time.  For flood advisories, DWRM needs to visit the AWLSs on an ad hoc basis.  For flood warnings, NECOC however has to rely on UNMA’s weather data and other open-sourced information 

Effectiveness - Outcome 2 Achievement

Moderately Satisfactory

Outcome 2: Effective use of hydro-met information for making EWs and long-term development plans

The overall TE rating is Moderately Satisfactory

Justification: 

% of population accessing improved CI & EWI (drought, flood & storm warnings) – Moderately Satisfactory - The percentage of population accessing CI / EWI has not been recently surveyed, nor was the TE able to sufficiently verify the figures.  Access to CI / EWI has improved significantly over the project lifespan. Following the integration and dissemination of CI / EWI through NECOC, the OPM has been able to issue a monthly National Integrated EWS (NIEWS) bulletin since December 2016.   The NIEWS bulletins are relayed through MoLG, down to district government.  UNMA generates the CI, which is added to by NECOC in terms of EW, and my MAAIF in terms of agricultural advice

Sector-specific policies, budgets & development plans that integrate CI – Moderately Unsatisfactory - Five national plans were assessed.  The NDPII, the MWE development plan and the UNMA strategic plan all consider the importance of creating an AWS network, accessing and utilizing the CI.  The National Policy for Disaster Preparedness was considered out of date (2010).  The MAAIF strategic plan doesn’t mention utilisation of CI.

Fifteen DDPs (DDPIIs 2015-20) were assessed, which all mentioned awareness of climate change and mainstreaming of climate change actions across sectors and into local programmes, however only a few described mainstreaming weather, CI and EW into planning and activities, with budgetary evidence weak.  This infers that SCIEWS was not effective in mainstreaming CI / EWI into local planning for this phase 2015-20, but it does provide evidence that a Phase II or follow-up project is required.

 

Efficiency

Efficiency

MS

Despite the issue of funds going to CIRDA and to Earth Networks, the project managed its funds well, including major contributions from UNMA and OPM in particular.

Efficiency Rating – Moderately Satisfactory

 

Relevance

Relevance

R

The project was relevant to national environmental and developmental priorities  and was in line with GEF / LDCF climate change priorities.  SCIEWS was designed in support of the National Development Plan (2010-14), National Disaster Risk Reduction & Management Policy (DRRM), National Climate Change Policy (NCCP, 2013/14), and NAPA (2007). 

Relevance Rating – Relevant

 

Implementation – Execution

Implemen-tation

Satisfactory

Project Implementation:  According to the given five categories (Implementing Agency - IA or Executive Agency - EA coordination & operational matters, partnership arrangements & stakeholder engagement, finance & co-finance, M&E systems (see next), and adaptive management (work planning, reporting & communications)

Overall Rating:  Satisfactory

Justification:  The implementation of most of the categories was satisfactory, with only minor short-comings (lack of M&E system, lack of a formal implementation agreement (apart from the project document) between partners).  Project implementation was fairly efficient and effective, although reporting (PIRs) was not of a high quality.  This was due to UNMA implementing the project, but the PMU reporting the project, there was a slight disconnect.

IA and EA Coordination & Operational Management

IA (UNDP) - Satisfactory

UNMA were satisfied with timely support from UNDP and the operational support provided by the PMU (under NIM), and noted that procurement had been 70% effective.  SCIEWS was noted for being responsive and adaptive to change / issues that arose, with risk being managed.  An example of adaptive management, would be the project no-cost extension of six months from 13 Dec 2013 until 30 June 2018, which allowed for the time-consuming procurement of equipment and operationalization to catch-up.

A PMU provided operational management support under National Implementation Modality, as UNMA were not accredited to take on the UNDP project directly.

IP (MWE / UNMA) – Satisfactory

Partnerships arrangements were mostly established for implementation, which were based on agreement of the approved prodoc with its co-financing letters of support from MWE, OPM, MAAIF, UCC as well as from UNDP.  The partnerships were administered and guided by the PB, its PTC and the PMU. 

The establishment of NECOC and their uptake of CI from UNMA and issuing the EW bulletins was of significant value.  Without this relationship, the project would not have been so successful.

Partnership Arrangements & Stakeholder Engagement

Operational partnership arrangements (institutional mechanisms and capacity) are the backbone for delivering new policies and services.  At national level, the following are assessed in the report:  UNMA, NMC, DWRM, OPM and their Department of Disaster Preparedness & Management, (DDPM), NECOC, MAAIF and their AEAS.  The strong working relationship between UNMA and OPM was of prime importance. 

For local partner engagement, there was a two-way flow of information from national to local level primarily involving the District Government & District Disaster Management Committee (DDMC), and the District Production Department (DPD).  This was facilitated by government institutional directives.

Finance

Disbursement of the GEF Trust Fund grant almost reached 100%.  Co-financing was considered to be good.

Adaptive management (work planning, reporting & communications)

Workplans and budgets were prepared.  Annual reports were produced by the PMU. PIRs were written.

Communications within the project were considered good, especially in having a PMU to support UNMA in the implementation of the project and liaise with UNDP.  What UNMA did not do was prepare any specific project implementation agreements with the other responsible parties apart from with OPM with whom an ‘administrative request system’ was used.  This meant that despite representation on the PB, the other parties tended to follow the lead of the chair (MWE) and the project coordinator (Director of UNMA), and in the case of the PTC, it was chaired by OPM.  Thus, the impression is that DWRM and MAAIF tended to be more passive partners in the project.  

Also, as the UNMA mandate was to deliver CI for national and district level consumption, it was left to OPM with the support of MoLG to deliver a mechanism to transfer CI / EWI to sub-county, parish and village level.

 

Monitoring & Evaluation

M&E

MU

M&E Systems – Design & Implementation

Overall quality of M&E – Moderately Unsatisfactory

Due to the short coming on not setting up an M&E system, feedback and adaptive management was slower than expected.  This impacted on the project in that it continued to have an equipment focus into year three, when other activities should have come more to the forefront earlier in the project cycle (e.g. training the next generation of weather analysts, bring MAAIF into the project)

M&E at Design - MU

SCIEWS didn’t set up a monitoring system at the start, thus it was unable to clearly see or report its results in a systematic way.

M&E Implementation - MU

Without an M&E system, the project was unable to report effectively with respect to a number of indicators, nor had the ability to feedback effectively on progress, such as dissemination of CI within the districts and its uptake.

 

Sustainability

Sustainability

ML

SustainabilityAccording to the four GEF risk categories (financial, socio-economic, institutional & governance and environmental), present status, and towards the future is assessed.

Overall Rating:  Moderately Likely

Justification:

Financial Risks to Sustainability – Moderately Likely  

UNMA is largely a public service that needs on-gong funds for maintaining its weather equipment and for maintaining a capability for CD analysis and CI production.  The state budget has increased significantly, which is supplemented by an annual request to development partners.  The UNMA strategic plan will need updating next year if not already, not least with MoFPED financing being on a three-year rolling system.

Concerning DWRM, the MoFPED is said not to provide much funding to it, however the MWE development plan does not bear this out.  In 2019/20, DWRM expect significant shortfall in budget in order to maintain all its new and upgraded stations provided by SCIEWS.  DWRM also needs funds for a flood modelling & forecasting unit.

Socio-Economic Risks to Sustainability - Moderately Likely

In the context of livelihoods, a promising start has been made, not least in updating the CI systems and the establishment of NECOC which are driving CI / EWI dissemination.  SCIEWS wasn’t designed to take the next step in ‘improving livelihoods’ and ensuring adaptation measures.  This now obviously needs to come in via the new Agricultural Extension & Advisory Service (AEAS) and new projects.

Institutional Framework & Governance Risks to Sustainability -Likely

UNMA – Long-range forecasting - Despite SCIEWS support, UNMA needs a further eight terabytes computer storage + processing power to run in-country seasonal climate models.  As part of efforts to decentralise met services, UNMA has officers proposed and budgeted for the four regional zones (Eastern, Northern, Western / SW, and Central), although they are not yet deployed.  This devolvement plan is expected to begin by the end of 2018, although the extent of regional services that can be provided is not clear.  The UNMA strategic plan indicates a need for 250 stations to cover a country of 248,000km2, which seems excessive.

DWRM has capacity issues with a limited capacity of staff to cope with the changing technologies; inadequate staff structure at central and zonal level; and inadequate equipment (e.g. flow measurement) at the zonal level.  The DWRM will close at Entebbe and be re-designed into a Water Resources Institute.  DWRM will relocate to Luzira.

UNMA’s NMC - Short-range forecasting – a number of the systems were not working when the TE visited, with added issues over their telecoms contract for data provision.  This in turn affects the safety of civil aviation.

Environmental Risks to Sustainability - Moderately Likely

DWRM are in the process of establishing a ‘water resources modelling & forecasting unit’, with a pilot consultancy to provide a flood prediction model and capacity, costs and equipment needs. 

Catchment planning under DWRM needs to be enacted and integrated.  MWE Water and Environment Sector Development Plan 2015/16 - 2019/20 Key priority 1 includes ‘promote catchment based integrated water resources management’.  The MWE plan indicates: UGX62.75b for integrated catchment-based water resources management (at the zonal level); UGX16m for developing & implementing 20 catchment-based management plans in four water management zones; and UGX 33.3m for operationalizing a flood management framework for Kasese, Awoja and Manafwa catchments.

 

Impact

Impact

Significant

Impact:  According to the three GEF categories (Significant, Minimal or Negligible), present status and towards the future

Rating: Significant

Justification:  SCIEWS, particularly with respect to UNMA has gone some way towards meeting the NAPA objectives including strengthening met services and expanding weather observation network.  UNMA is at the stage of a major upgrading of CD systems installation and improving CI modelling / analytical requirements.  The beneficiaries of CI such as MAAIF and NECOC are at an early stage in the further packaging and dissemination of CI / EWI, thus much remains to be achieved.  SCIEWS has significantly raised climate information on the political agenda, and in understanding that such information is required, as a starting point for integrated climate planning at all levels, including catchment and EWSs.  SCIEWS has demonstrated the need for further enhanced state and donor inputs.


Recommendations
1

1. The next UNMA strategic plan needs to be prepared with inclusion of a longer-term forecast on state budgetary needs for a set minimum level of required public services

2

UNMA to assess their ability to provide tailored CI on a district-basis, to be piloted in areas which are disaster prone or of strategic agriculture importance for example.  This could be piloted under another project and provide an opportunity to train further meteorology analysts

3

3. UCC and UNMA need to ensure that NMC becomes fully operational, with a new telecoms agreement for unlimited high-speed fibre-optic data transmission and its systems for short-range weather forecasting for aviation debugged. [UCC / UNMA]

4

4. The development of a new GCF-funded project should be expedited, with the inclusion of piloting dissemination of CI to farmer level 

5

5. An EWS for flood response is needed.  A management decision is required on the level of security and local awareness needed for key AWLSs to be automated with telemetry 

6

6. Catchment-based water resources planning is needed to mitigate flooding.  The technical capacity of DWRM and its zonal offices needs to be strengthened.  Catchment-based plans are part of the MWE plan, however a donor project also should be sought to strengthen this

7

7. The main lines of agri-climate information analysis and transfer need to be assessed and strengthened, so that CI is better utilized 

1. Recommendation:

1. The next UNMA strategic plan needs to be prepared with inclusion of a longer-term forecast on state budgetary needs for a set minimum level of required public services

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

2. Recommendation:

UNMA to assess their ability to provide tailored CI on a district-basis, to be piloted in areas which are disaster prone or of strategic agriculture importance for example.  This could be piloted under another project and provide an opportunity to train further meteorology analysts

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

3. Recommendation:

3. UCC and UNMA need to ensure that NMC becomes fully operational, with a new telecoms agreement for unlimited high-speed fibre-optic data transmission and its systems for short-range weather forecasting for aviation debugged. [UCC / UNMA]

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

4. Recommendation:

4. The development of a new GCF-funded project should be expedited, with the inclusion of piloting dissemination of CI to farmer level 

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

5. Recommendation:

5. An EWS for flood response is needed.  A management decision is required on the level of security and local awareness needed for key AWLSs to be automated with telemetry 

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

6. Recommendation:

6. Catchment-based water resources planning is needed to mitigate flooding.  The technical capacity of DWRM and its zonal offices needs to be strengthened.  Catchment-based plans are part of the MWE plan, however a donor project also should be sought to strengthen this

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

7. Recommendation:

7. The main lines of agri-climate information analysis and transfer need to be assessed and strengthened, so that CI is better utilized 

Management Response: [Added: 2018/11/27]

Agree

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org