Annual Evaluation of Project: Sustainable Development and Integrated Water Management

Report Cover Image
Evaluation Plan:
2017-2021, Saudi Arabia
Evaluation Type:
Mid Term Project
Planned End Date:
10/2019
Completion Date:
11/2019
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
40,000
  1. The objective of this program is to initiate a systematic process of capacity development to help in sustainable development of water resources and management of water-related affairs in the Kingdom to ensure permanent and sufficient supply. In order to make a transition to sound water management mode, it is important to strengthen the technical and organizational capacities of the MEWA to deal with the triple challenge of water exploitation and distribution (operational side), research for additional resources and cutting-edge technologies to satisfy the increasing demand (research side), and rigid control and administration of all water-related aspects (control side). Second, a sound information base covering data on groundwater availability, quality, withdrawal, and usage is about to be put in place
  2. The specific objectives of this evaluation are to assess progress towards achieving project targets, assess the project appropriateness of the project design and its relevance and effectiveness and effectiveness, efficiency and sustainability in achieving results. This evaluation focuses more on process and results of first 18 months implementation and draw on emerged issues and opportunities to enable adaptive management.
  3. Key achievements (among many others)
    • Significant contribution to enhance water management capacity at MEWA
    • Led the development of policy paper for the upcoming G20 summit in KSA titled “Fostering Sustainable and Resilient Water System Globally”.
    • Helped the ministry to report on SDG 6 (sustainable water)
    • Developed water supply-demand model for the first time in Saudi,
    • Established a groundwater model for KSA
  4. Evaluation findings
    • The project overall objectives and outputs are aligned to the MEWA needs and priorities, vision 2030 and UNDP priorities.
    • Weak design of the project document (results framework, governance structure and risk identification and management).
    • A moderately satisfactory progress in meeting its expected results. The review of the project achievements indicates generally a strong focus on activities as opposed to developmental results (i.e capacity building).
    • The current management arrangements are ineffective. These are largely the result of ambiguities in the original project design (the Project Document).
    • Limited project management capacities in the project.
    • The project is largely reliant on external funding sources to implement its activities and no clear resource mobilization plan outlining funding alternatives – should the NTF funding fails. 
    • Key project outputs are under-resourced, mainly the NCWRS and Water Extensional Education Center.
    • There are key concerns over the sustainability of the program results, especially in terms of skills and knowledge transfer from experts to the MEWA staff, and MEWA ability to run the business beyond the project.
  5. Key recommendations
    • Undertake a substantive review of the project document
    • Identify capacity needs and develop a capacity building plan
    • Re-structure the project management unit with an intent to enhance the project management capacity and streamline decision making process.
    • Re-activate the project board and project coordination group (currently called steering committee)
    • UNDP to support the project with communication expertise.
    • Upgrade the software and hardware assets to meet the modelling technical requirements
    • Assess the project financial needs and develop a resource mobilization plan
    • Review the project resourcing strategy and ToRs of all experts and address the deficit
    • Design fit-for-purpose reporting templates with a greater focus on the M&E framework – including the performance indicators.

Share

Title Annual Evaluation of Project: Sustainable Development and Integrated Water Management
Atlas Project Number: 00107640
Evaluation Plan: 2017-2021, Saudi Arabia
Evaluation Type: Mid Term Project
Status: Completed
Completion Date: 11/2019
Planned End Date: 10/2019
Management Response: Yes
UNDP Signature Solution:
  • 1. Resilience
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.4.1 Solutions scaled up for sustainable management of natural resources, including sustainable commodities and green and inclusive value chains
SDG Goal
  • Goal 12. Ensure sustainable consumption and production patterns
SDG Target
  • 12.2 By 2030, achieve the sustainable management and efficient use of natural resources
Evaluation Budget(US $): 40,000
Source of Funding: Government Cost Sharing
Evaluation Expenditure(US $): 31,500
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: No
Key Stakeholders:
Countries: SAUDI ARABIA
Lessons
1.
  • Adaptive management and the project cycle should be clearly understood. Absence of adaptive and agile project management approach leads to further complicate the barriers and missing the change opportunities. The inception phase is the first opportunity to review the current status of the project and to determine whether circumstances have changed against the assumptions made in the Project Document. The inception phase should have the confidence and authority to significantly challenge the project’s design and not to accept it as a fait accompli. The role of an inception phase that could have been utilised to allow for validating the project design prior actual implementation starts. It would have helped addressing the deficiencies in the project design. Such a project needed a defined inception phase at start up to review the design elements, engage stakeholders. Any changes to the programme strategy, management arrangements, monitoring framework and participation of stakeholders should be documented in an inception report, which should be endorsed by the management committee overseeing the development of the programme/project.
  • Project governance and management arrangements are critical to the successful outcome of a project. Much of the confusion surrounding the management arrangements in the project appear to have stemmed from imprecise instructions in the Project Document. Deficiencies in project governance design lead to severe delivery risks. This was evidently the case with this project where governance was largely ineffective and highly influenced the decision making and day to day business.
  • Projects need to balance the focus on activities versus the developmental results. Generating high quality products and services may not be enough, unless achieving the ultimate intended outcome (i.e capacity building). In case of this project, there are significant achievement mainly studies, analyses, information, and training events. There are indispensable deliverables but they also remain as information products, unless stakeholders uptake these information, used it and build the capacity, the long-term impact of the project will be limited.

Findings
1.

5. Findings and conclusions

5.1 Appropriateness (project Design and relevance)

5.1.1 Key evaluation questions 1. How does the project relate to the needs of the MEWA, and its strategic plan? 2. How well the program is designed? 

Relevance scale -1. Not relevant (NR)

Project design scale - Moderately Unsatisfactory (MU)- significant shortcomings

Overall conclusions: Project relevance - Relevant -The project overall objectives and outputs are aligned to the MEWA needs and priorities, vision 2030 and UNDP priorities. Project design -Moderately Unsatisfactory -Weak design of the results framework, governance structure and risk identification and management. And the process was cumbersome and the project document was limited to be used as a “blue print” for the Implementation Team. 

5.1.2 Project relevance 

The initial project design including 5 key outputs (see above) have been reasonably justified and aligned to the MEWA needs and priorities. In principle, the objective of the project and its outputs remain as relevant today as when the project was conceived. MEWA genuinely requires capacity building outcomes in the areas covered by the project outputs, and the UNDP project was designed to deliver these outcomes. The National Water Strategy (NWS) has been recently developed and aims at achieving sustainable water sector, safeguarding the natural resources and environment of the Kingdom and providing cost-effective supply and high-quality services. Efficiency contributes to economic and social development. Derived from the sector’s five strategic objectives, the NWS defined ten strategic programs and initiatives. The 5 key outputs plus the additional 2 new suggested correspond with those programs defined in the NWS. For instance, output #3b helps to achieve program #1 under the NWS through supporting ground water use regulations and enforcement. Similarly all other outputs contribute directly to programs 2,3 and 4 under the NWS. 

The project outputs are also highly relevant to achieve the Vision 2030 goals the Kingdom of Saudi Arabia and, specifically, in this case, to meet all of the SDG goals especially as pertains to this project in terms of water in Goal 6: Clean Water and Sanitation, Goal 11: Sustainable Cities and Communities, Goal 12: Responsible Consumption and Production and, Goal 13: Climate Action.UNDP has identified three programme priorities for the new Country Programme Document (CPD). These are aligned with the national priorities articulated in Vision 2030, the tenth national development plan, and the UNDP strategic plan. They address areas of diversification and growth; employment and vulnerability; access to efficient public services; and non-oil natural resources management. The project directly relevant to the pillar three and helps manage the most significant non-oil natural resource, water. Relevance of the project outputs to the needs of the MEWA is essential for rationalizing the project, however, a successful project design is the one that selects the right activities and bound them with a coherent governance and robust monitoring and evaluation. While the majority of activities are relevant, it is noted that the hydrothermal energy component of the project is irrelevant to the MEWA mandates and priorities. The project stakeholders have suggested removing this component as part of the re-alignment process. 


Tag: Natural Resouce management Water resources Relevance Monitoring and Evaluation Policies & Procedures Results-Based Management Capacity Building

2.

5.1 Appropriateness (project Design and relevance) - Continued

5.1.3 Project design 

The project was designed in 2017, which arguably puts it as a “second phase” to the previous UNDP project that support the development of a National Water Strategy (NWS). At the time the NWS was not fully finalized, which meant that the project formulation was concurrent to the finalization of the NWS.Overall design of M&E framework is unsatisfactory, the project lacks for meaningful and Specific, Measurable, Achievable, Realistic, and Time-bound (SMART) Key Performance Indicators (KPIs) that would obtain a robust evidence for the project effectiveness and efficiency. The defined KPIs in the project document are insufficiently SMART, they are repetitive, that is they were effectively tracking the same outputs. Some of the indicators were irrelevant, for instance tracking the percentage of achieving data collection (for activity 2.6 in the original project document) doesn’t provide any insight on the development of the master plan and its effectiveness. Most of the indicators, if not all, are output delivery-based, and lack for insights for the ultimate outcome anticipated in the project (i.e capacity building). The phrasing of some of the indicators was not precise enough and in a lesser project it might have been open to interpretations. 

In the event that the project had not invested sufficiently in data collection for the right indicators, it is important to re-design the project results framework (section 5 of the project document) and establish baselines and realistic targets. 

Governance is the means for achieving direction, control, and coordination that determines the effectiveness of management. In this case effective management can be taken to mean ensuring that the interaction of people within the project are organized in such a way that achieves capacity building outcomes through implementing participatory delivery model and effective decision making. The design of the governance structure, as in the project document, is unsatisfactory and presents a significant challenge to the project delivery. The project management has been unable to influence change in directorates over which the it has very little control and whom may have widely differing agendas and means of measuring success. Below figure represents the project governance model as presented in the project document. It is noted that MEWA is the Implementing Partner with UNDP Implementation Support Services. The model doesn’t separate between coordination platform and decision-making forums. This required projects to work across a multiple of directorates and outputs and in areas where there are complex external dependencies, for example seeking funding from the NTF. Furthermore, while changes to the scope of work is often a technical challenge but it has o be accompanied with a rigorous process and coherent governance. 

The governance model doesn’t define the roles and responsibilities between key personnel engaged in the project management, namely the CTA, project manager and NPC. In fact the project management function is somehow scattered and missed out among these players. As well, the reporting lines between project management and output managers are also not clear, this resulted in creating multiple reporting burden for the output managers to each of the project manager, NPC and CTA, and in some cases, contradicting guidance from three project management personnel to the output managers. 

The UNDP experts fill senior positions at the MEWA, namely 6 out 8 directorates are managed by UNDP experts. While this reflects the significance of UNDP experts contribution to the project and broader MEWA’s work, it, however, creates two legitimate concerns for the project to address: 

• A significant deviation from the original technical duties that these experts were meant to deliver on, and doing more managerial work at the expense of utilizing their technical competencies. • The sustainability of these senior positions beyond the project timeframe, assuming the expert contracts will no longer be extended after the project, which means that 6 directorates will be vacated from the director role with no replacement strategy in place. 


Tag: Relevance Human and Financial resources Monitoring and Evaluation Operational Efficiency Programme/Project Design Project and Programme management Results-Based Management Risk Management Sustainability Theory of Change Capacity Building Coordination Data and Statistics

3.

5.2 Effectiveness

5.2.1 Key evaluation questions

1. To what extent are the expected activities of the project being achieved in the first 18 months?

2. How is the project effective in achieving its expected outputs/outcomes?

Effectiveness scale -4. Moderately Satisfactory (MS)- moderate shortcomings

Overall conclusions  - Project effectiveness - Moderately Satisfactory -• A moderately satisfactory progress in meeting its expected results. Anticipated activities for the first 18 months have been implemented, with portion of shortcomings. • The review of the project achievements indicates generally a strong focus on activities as opposed to developmental results (i.e capacity building). • The project has not established clear understanding of the capacity needs yet, and lacks for clear capacity building strategy to achieve the outcome • Inadequate communication of project achievements to MEWA decision makers and broader stakeholders

The project aims at strengthening and optimizing the technical and organizational capacities of the MEWA with respect to water resources and water supply management of the Kingdom. With the increase in tasks attributed to the MEWA and the concomitant increase in the portfolio, the MEWA personnel must be trained to meet the new challenges. The cooperation with UNDP and its provision of international experts for project work and capacity development is expected to ensure that the capability of the MEWA meets the highest standards to carry on all tasks at the best possible level.

The review of project achievements indicates a moderately satisfactory progress in meeting its expected results from the beginning of the project until the time of this evaluation. Anticipated activities for the first 18 months have been largely implemented, with some shortcomings. The project has made significant contribution to enhance water management capacity at MEWA, that was evidently lacking prior project implementation commences. The project was able to attract water management expertise who helped to finalise the strategic plan and its indictors. Moreover, the project has established conceptual frameworks and policy briefs on water economics and how it works in KSA, environmental economy and other key water management concepts for the first time at the ministry. The project led the development of policy paper for the upcoming G20 summit in KSA titled “Fostering Sustainable and Resilient Water System Globally”. The paper proposes two major outcomes, the first being a Roadmap for the establishment of an International Water Organization, and the second, a G20 Action Plan on Water Management. It is worth mentioning that it is the only paper, among other 70 accepted by the G20 secretariat, that was developed in house (i.e from within Government).

The project helped the ministry to report on SDG 6 (sustainable water), this included scientific analysis of the SDG indicators to be reported as part of the Voluntary National Report. The project developed water supply-demand model for the first time in Saudi. The model is comprehensive enough to include all related parameters influencing water supply and demand, and predict changes in supply and demand over the coming 30 years. Modelling future water supplies and demand supports government to undertake coordinated, long-term water supply planning for the sustainable development across the country. The modelling started at the country level and then cascaded down to the local level covering 1517 centres across the Kingdome, this means the ministry has developed a good understanding of the water supply-demand status and future at the local level, including gaps and potential strategic solutions. Figure 05 below shows the supply-demand projections at the national level. The model now provides an intimate knowledge of all facilities that serve water abstraction, distribution, and supply to the customers.

The project has developed groundwater model that assesses the status of ground water aquifers, quantifies groundwater recharge through considering rainfall, infiltration, and evapotranspiration and predicts the effects of groundwater abstraction, for each of the major groundwater systems, an individual groundwater model has been developed during the past decade.The project supported a detailed hydrological and engineering study of dams to evaluate the operational and maintenance plan as well as evaluation of sediment accumulation. These studies aim at assessing existing status of the dams, and most importantly facilitate selection of new potential sites for new dams. The selection of new sites has been informed by extensive research and modelling with an aim to maximize benefits from the dams.The project implements the so called “water regulation” initiative. The initiative is funded by NTP (SRY 280 Million, and aims at installing smart meters on ground water sources, these meters will be automatically connected with a central data base. Priority was given to 1700 ground water wells that represent more than 80% of the ground water consumption by agricultural, commercial and industrial businesses. The project team has also drafted regulations and licensing criteria for drinking water, which in turns determine the amount of water abstractions from wells. The project is also in process of automating all data sources and communication protocol for licensing and other data bases (SAR 23 million funding from the NTP). Nowadays, licensing drinking water industries happens completely online. 20% of the automation process is estimated to be delivered and the rest is underway.As far as enforcement is concerned, the project started a process of tracking the machinery that dig water well in order to know their whereabouts and control their movements.As far as the control room is concerned, the project team has brought together raw data and information from across three software used at MEWA (IPS, info-mat and hydro-manager) into one unified data base available at the intranet. The project has recently secured funding of SAR 900K from the core public funding to start with a small control room that makes live data available. The project has also secured a piece of land next to existing MEWA premise on which the new centre will be constructed. 

The project had little contribution towards building the capacity of MEWA to implement the NWS (considering the broad definition of capacity as mentioned above). When considering the level of exiting MEWA capacities without the project expertise, the MEWA would not have been able to achieve what has been achieved in the first 18 months (see below the achievements). There is clear evidence of the contribution of the project to develop capacities of MEWA on matters related to implementation of the NWS priorities. For example, for the first time ever, the project delivered dynamic supply-demand model that forecasts changes in water supply and demand over the next 30 years. Other examples are detailed below.The project has no clear capacity building strategy implemented in the project. There have been few training activities that were challenged with people motivations to attend the trainings. However, trainings are just one component of the capacity building sought to be achieved by the project.

Particularly output 2 of the project provided a good practice in delivering in job training for new MEWA staff. This involves heavy engagement between the experts and new staff, and ongoing brainstorming sessions. The director water resources department is a strong believer in such kind of capacity building and a champion of making it actually happens. 


Tag: Drinking water supply Water resources Effectiveness Communication Knowledge management Policies & Procedures Results-Based Management Capacity Building

4.

5.3 Process

5.3.1 Key evaluation questions

1. How effective the project governance, stakeholders engagement, delivery mechanisms and decision making processes adopted by the program?

Process scale- 3. Moderately Unsatisfactory (MU) -significant shortcomings

Overall conclusions -Project process -Moderately unsatisfactory - • Limited application of adaptive management approach to capture emerging opportunities and overcome faced barriers. For instance, the project recognized the need to re-design the scope and management arrangements, but had not implemented the review process according to standard project management requirements. • The current management arrangements are ineffective. These are largely the result of ambiguities in the original project design (theProject Document) and ad hoc arrangements in the different components of the project. • Six project experts are performing a general director role at MEWA, in a director capacity, the experts’ time is dedicated to undertake managerial tasks at the expense of the technical input. In addition, the sustainability risk that this model brings is not addressed in project delivery and monitoring. • The project teams have limited knowledge and experience in project management basics and skills that are intrinsic to the successful project delivery, such as project monitoring, reporting, risk management, and outsourcing consulting services.


Tag: Effectiveness Project and Programme management Results-Based Management

5.

5.3 Process

5.3.1 Key evaluation questions (Continued)

5.3.3 Management approach

This project is administered from the Head Office of the Ministry of Environment, Water and Agriculture (MEWA) in addition to the UNDP’s Implementation Support Services are provided from the Country Office. As indicated under section 5.3.1, the project has been suffering from poor governance design. The project steering committee was active for some time and then was put on hold, there has been less appetite to participate in the steering committee. The steering committee brings together the project management team and directors of directorate and UNDP quality assurance team. The committee however has no clear mandate, it looks as though more of coordination platform and doesn’t involve “steering” as opposed to decision making.The current arrangements are largely the result of ambiguities in the original project design (the Project Document) and ad hoc arrangements in the different components of the project. It is evident that these arrangements have not worked effectively given the chaotic circumstances in the first year and a half of project execution. These arrangements have resulted in inefficiencies. As a result of the poor governance design (section 5.3.1), the decision-making process has been mostly made individually by the project management team members (NPC, PM, CTA) with limited effective participatory decision-making approach. Roles and responsibilities among project management team are not clear enough, it is noted that duties clearly overlap. Project monitoring and reporting function is an example where project management duties overlap, and as a result miss-coordinated, between the project management members. While this function is typically the responsibility of the project manager, however the project document assigns this to the CTA. The project document indicates “A Chief Technical Advisor will be appointed to manage his/her own component as well as oversee all technical project activities, monitor progress and report to both MEWA and UNDP via quarterly and annually progress reports”. This resulted in deviating the focus of the CTA from technical work into major administrative work. 

Coordination and engagement in this project are seen at three levels, a) Technical level: this involves coordination of the project activities level and make sure that project teams are aware of each other’s work, identify efficiencies and avoid duplication. b) At the project operation level: this includes coordination among the members of the project management team (NPC, CTA & PM), this coordination help to achieve participatory decision making.c) At the strategic level (project board): where seniors of the MEWA and UNDP meet to guide the project strategically and make strategic decisions such as project change of scope, approving work plan and budget, etc. 

The “steering committee” has facilitated the coordination at the first level (i.e the technical level) for sometime, and there is limited coordination at both second and third levels. Particularly the project board has not been activated at all. The project teams have limited knowledge and experience in project management basics, skills, such as project monitoring, reporting, risk management, and outsourcing consulting services, are intrinsic to the successful project delivery. The UNDP experts fill senior positions at the MEWA, namely 6 out 8 directorates are managed by UNDP experts. While this reflects the significance of UNDP expert’s contribution to the project and broader MEWA’s work, it, however, creates two legitimate concerns for the project to address.  

• A significant deviation from the original technical duties that these experts were meant to deliver on, and doing more managerial work at the expense of utilizing their technical competencies. • The sustainability of these senior positions beyond the project timeframe, assuming the expert contracts will no longer be extended after the project, which means that 6 directorates will be vacated from the director role with no replacement strategy in place.

The review indicates that the day-to-day management of the project was much activity-based as opposed to be more results-based (RBM). This focus on activities has been driven by the design that is much activity-based. As a result, the project technical resources (CTA and experts) have been driven towards more administrative tasks. The project experts are assigned to multiple tasks, often outside their original mandate as defined in their terms of reference, and this potentially causes delay in delivering project activities. 

There have been some ambiguities concerning the recruiting process of the project experts, such as: • On what basis the number of experts for each output has been determined • The extent to which the experts background and experience matches the required tasks • The terms of reference of the experts are rather generic and leave rooms for subjective interpretations • The contract modality for experts engagement, full time jobs versus short term assignments. 

The MEWA senior management is unaware of the UNDP working modality and services that it could provide to the project, including UNDP procurement and recruitment advantages. In fact, it has been noted by interviewees of the MEWAthat UNDP (referring to the country office) had little contribution to the project decision making and daily business. UNDP’s comparative advantage must be viewed through the complex nature of the project and the difficulties it has encountered particularly related to the management arrangements and governance and balance the degree of the autonomy and ownership vs quality assurance and support services. 


Tag: Monitoring and Evaluation Oversight Partnership Project and Programme management Results-Based Management Civil Societies and NGOs Capacity Building

6.

5.4 Efficiency

5.4.1 Key evaluation questions

1. How efficiently have the project resources been turned into results?

Efficiency scale - 4. Moderately Satisfactory (MS)- moderate shortcomings.

Overall conclusions - Project efficiency -Moderately satisfactory - • The project is largely reliant on external funding sources to implement its activities and no clear resource mobilization plan outlining funding alternatives – should the NTF funding fails. • Key project outputs are under-resourced, mainly the NCWRS and Water Extensional Education Center. • The project failed to obtain the required hardware and software to undertake the highly technical modelling work. • The project is going through s budget deficit of ~$1 mil, mainly due to hiring more experts than originally planned to. 

This Section presents findings on the efficiency of the project that is a measure of the productivity of the project intervention process. It reviews to what degree achievements derived from an efficient use of financial, human and material resources.The total budget for the project was set out in the Project Document and gives a total budget of US$ 24,850,429.00, equivalent to SAR 93,189,108.75. The budget is largely allocated to cover project teams expert cost, with few opex resources available for the Water Extensional Education Center and scholarship program. The opex resources required to operationalise the project activities are to be funded either by MEWA recurrent public funding or by the National Transformation Plan (NTP) under vision 2030. The NTP aims achieving Governmental Operational Excellence, Improving Economic Enablers, and Enhancing Living Standards through:Accelerating the implementation of primary and digital infrastructure projects. • Engaging stakeholders in identifying challenges, co-creating solutions, and contributing to the implementation of the program’s initiatives. 

Particularly output #3 is highly dependent on funding from the NTP. The funding process seems quite challenging and demanding from the project team perspective. So far the project team secured SAR 280 Million from the NTP to implement the “water regulation” initiative. The project is also in process of seeking extra funds from NTP to implement the automation of all data sources and communication protocol for licensing and other data bases (SAR 23 million), of which the project has been successful in securing seed funding to outsource consulting service to develop a full detailed business case and project scope to satisfy the NTP requirements.Some of the project outputs are still largely under-resourced to operationalise its agenda, for example the NWCRS have only 4 personnel assigned to deliver ambitious research agenda with little chances of outsourcing research activities due to lack of funding. 

The project teams are also challenged to find proper office spaces and equipment. Some of the project experts have been, and some are still, using their personal laptops including the project CTA. Others are struggling to find a convenient office to work at. The project experts were challenged to obtain hardware and software required to undertake the work. Modelling work often requires sophisticated software and hardware with relatively high specifications. The project experts have set up the specifications and handed them over to the procurement at MEWA, unfortunately the specifications were changed prior purchase without experts being consulted.


Tag: Efficiency Resource mobilization Human and Financial resources Project and Programme management Data and Statistics

7.

5.5 sustainability

5.5.1 Key evaluation questions

2. What are the probabilities that the project achievements will continue in the long run?

Sustainability scale - 3. Moderately Likely (ML) - Moderate risks 

Overall conclusions - Project sustainability - Moderately Likely- • There are key concerns over the sustainability of the program results, especially in terms of skills and knowledge transfer from experts to the MEWA staff, and MEWA ability to run the business beyond the project. • The project document has no discussion on sustainability and how it should be secured through project results. • The outcomes of groundwater regulations and enforcement component will likely sustain beyond the project. • The MEWA sees the project valid and relevant and therefore participate in and own the project activities. • The financial sustainability of the project is uncertain in light of the funding dependencies and ambiguities of the project resources gaps and mobilization plan. 


Tag: Water resources Sustainability Knowledge management Ownership Policies & Procedures Country Government Capacity Building

Recommendations
1

Recommendation 1. Undertake a substantive review of the project document

The review of the project would mainly aim at: 

- Documenting the change of scope and its subsequent implication on other project aspects such as budgeting.

- Developing a new governance model and management arrangements and address the gaps identified in this evaluation, mainly as related coordination and decision making mechanisms. - Re-design the M&E framework, including development of new SMART indicators that would obtain a robust evidence for the project intent (i.e capacity building) effectiveness and efficiency.

- Develop a new risk log that identifies emerging risks and mitigation strategies.

- Review the project outputs needs of experts and suggest resources allocations to each output including advice on the contracting modality (e,g full time expert vs short time assignment). 

2

Recommendation 2. Identify capacity needs and develop a capacity building plan

Despite the delay, the project team has recently developed a ToR to assess the capacity needs, this is now going through tendering process. It is important that this component is looked through carefully by identify the individual and institutional capacity needs in a systematic approach, and most importantly to be followed by a capacity building strategy that provides wide range of capacity building activities customised to address the priority needs, including means that ensure transfer of skills and knowledge such as mentorship or shadowing happening between recruited experts and MEWA staff.

The evaluation indicated that the sustainability of project achievements should be ensured, particularly due to the strong national ownership of these achievements. Most achievements are already institutionalized, which is an excellent first step toward sustainability. However, the project needs to define its exit strategy after the project ends including extension scenarios and how the MEWA would be able to run its business beyond the project.

3

Recommendation 3. Re-structure the project management unit with an intent to enhance the project management capacity and streamline decision making process

This should include supporting the PMU with strong project management capacities, clarifying project management duties clearly between the PMU members (PM, CTA, and NPC), and enabling the deputy minister to perform higher strategic role in the project decision making and governance. 

In order to fill the project management gap, it might be feasible to create a new position of a National Project Director (NPD) who reports directly to the deputy minister (the NPC) and directly manage the project manager and the CTA. The project director will need to be highly competent in applying UNDP project management framework and strong coordination and communication skills, the project director will have an overall responsibility for project-managing the entire project, implementing governance arrangements and coordinate the project delivery across the project teams and UNDP. This creates a hierarchical reportinglines in the project and enables the CTA and other technical experts to be more dedicated towards the technical aspects. This suggestion needs, however, to be investigated prior adoption, and after the understanding of the budget implications. 

Coordination and engagement in this project are seen at three levels,

d) Technical level: this involves coordination of the project activities level and make sure that project teams are aware of each other’s work, identify efficiencies and avoid duplication. e) At the project operation level: this includes coordination among the members of the project management team (NPC, CTA & PM), this coordination helps to achieve participatory decision making at the operational level. f) At the strategic level (project board): where seniors of the MEWA and UNDP meet to guide the project strategically and make strategic decisions such as project change of scope, approving work plan and budget, etc.

4

Recommendation 4. Re-activate the project board and project coordination group (currently called steering committee)

This project board is critically crucial in influencing the strategic directions of the project. The board is a strong governance platform to make on consensus-based management decisions for a project, provide strategic guidance and help to overcome risks. It is important to convene the project board with the same composition as described in the project document, at least, twice a year. The project board is particularly needed to oversee the implementation of the recommendation in this evaluation. 

It is also recommended that the project steering committee ToR is reviewed with an intent to make this a coordination platform only and no decision making, this may require to call it as Project Coordination Group (PCG) for technical coordination, harmonization, communication and linkages. The PCG should meet, at least, on a monthly basis to ensure proper coordination happening. 

5

Recommendation 5. UNDP to support the project with communication expertise

 It is obvious that there is inadequate communication of project achievements to MEWA decision makers and broader stakeholders. It is understood that UNDP CO has recently established a communication capacity to support various programs, and it is highly recommended that a communication specialist supports the PMU to communicate the project achievements appropriately and effectively. 

6

Recommendation 6. Upgrade the software and hardware assets to meet the modelling technical requirements

The project failed to obtain the required hardware and software to undertake the highly technical modelling work, as this is essential to undertake the modelling work on ongoing basis, it is recommended to invest in such a foundational basis to deliver high quality product and upgrade the already purchased software and hardware, and if required, buy a new ones. 

7

Recommendation 7. Assess the project financial needs and develop a resource mobilization plan

As noted in the evaluation, the project offers relatively limited resources comparing to the scale of the activities. The project is highly dependent on external funding, in some cases, non-recurrent public funding. It is recommended to assess the funding gaps, map funding opportunities and develop resource mobilisation plan including scenarios for securing the required funding. 

8

Recommendation 8. Review the project resourcing strategy and ToRs of all experts and address the deficit

It is recommended to undertake a holistic review of all experts ToRs to be more driven by the needs under each output, and balance the experts allocations to different outputs as well as review the expert engagement modality including short-term assignment in case if the sought deliverables don’t necessarily need a full time-job.

In light of these proposed changes in experts engagement modalities and other changes in resourcing strategy, the budget deficit need to be addressed as well.

9

Recommendation 9. Design fit-for-purpose reporting templates with a greater focus on the M&E framework – including the performance indicators

The monitoring template used by the project includes a long list of questions - in addition to the reporting on performance indicators - that need to be answered to complete a semi-annual monitoring report. It is too time consuming, cumbersome to complete and the result is that it does not provide accurate and timely information on how the project is progressing. There is a need to review the reporting template, to shorten it and focus on the performance framework with the set of indicators as the central part to measure how well the project is progressing toward the achievement of its expected outputs and outcomes.

On that front, it is important to guide the project management to deliver fit-forpurpose reporting. For example, developing a 1-page dashboard reporting template targeting decision makers. 

Management Response Documents
1. Recommendation:

Recommendation 1. Undertake a substantive review of the project document

The review of the project would mainly aim at: 

- Documenting the change of scope and its subsequent implication on other project aspects such as budgeting.

- Developing a new governance model and management arrangements and address the gaps identified in this evaluation, mainly as related coordination and decision making mechanisms. - Re-design the M&E framework, including development of new SMART indicators that would obtain a robust evidence for the project intent (i.e capacity building) effectiveness and efficiency.

- Develop a new risk log that identifies emerging risks and mitigation strategies.

- Review the project outputs needs of experts and suggest resources allocations to each output including advice on the contracting modality (e,g full time expert vs short time assignment). 

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Review each activity, activity group, and output. Change scope, modify it, and define new ones. All directors and output managers interviewed, everything rearranged, additional personnel needs noted and calculated.
[Added: 2019/12/15] [Last Updated: 2020/01/25]
Project Chief Technical Advisor 2019/12 Completed All actions completed, personnel and project needs all taken into consideration, awaiting final approval by National Project Coordinator and Project Board, due to meet 2 February History
2. Recommendation:

Recommendation 2. Identify capacity needs and develop a capacity building plan

Despite the delay, the project team has recently developed a ToR to assess the capacity needs, this is now going through tendering process. It is important that this component is looked through carefully by identify the individual and institutional capacity needs in a systematic approach, and most importantly to be followed by a capacity building strategy that provides wide range of capacity building activities customised to address the priority needs, including means that ensure transfer of skills and knowledge such as mentorship or shadowing happening between recruited experts and MEWA staff.

The evaluation indicated that the sustainability of project achievements should be ensured, particularly due to the strong national ownership of these achievements. Most achievements are already institutionalized, which is an excellent first step toward sustainability. However, the project needs to define its exit strategy after the project ends including extension scenarios and how the MEWA would be able to run its business beyond the project.

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Will be done after Phase of Applus Project.
[Added: 2019/12/15] [Last Updated: 2020/06/11]
Project Manager 2020/06 Completed capacity assessment completed History
Parallely, low-level starter courses are developed
[Added: 2019/12/15] [Last Updated: 2020/11/16]
Consultancy Firm/Project CTA 2020/08 Completed Courses were delayed because of COVID but have now been approved by National Project Coordinator History
Define its exit strategy
[Added: 2019/12/15] [Last Updated: 2020/11/16]
CTA/PM 2020/12 Completed The new CTA is aware of the needed exit strategy and is working on it History
3. Recommendation:

Recommendation 3. Re-structure the project management unit with an intent to enhance the project management capacity and streamline decision making process

This should include supporting the PMU with strong project management capacities, clarifying project management duties clearly between the PMU members (PM, CTA, and NPC), and enabling the deputy minister to perform higher strategic role in the project decision making and governance. 

In order to fill the project management gap, it might be feasible to create a new position of a National Project Director (NPD) who reports directly to the deputy minister (the NPC) and directly manage the project manager and the CTA. The project director will need to be highly competent in applying UNDP project management framework and strong coordination and communication skills, the project director will have an overall responsibility for project-managing the entire project, implementing governance arrangements and coordinate the project delivery across the project teams and UNDP. This creates a hierarchical reportinglines in the project and enables the CTA and other technical experts to be more dedicated towards the technical aspects. This suggestion needs, however, to be investigated prior adoption, and after the understanding of the budget implications. 

Coordination and engagement in this project are seen at three levels,

d) Technical level: this involves coordination of the project activities level and make sure that project teams are aware of each other’s work, identify efficiencies and avoid duplication. e) At the project operation level: this includes coordination among the members of the project management team (NPC, CTA & PM), this coordination helps to achieve participatory decision making at the operational level. f) At the strategic level (project board): where seniors of the MEWA and UNDP meet to guide the project strategically and make strategic decisions such as project change of scope, approving work plan and budget, etc.

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Detailed review of positions described in contract. Detailed review of other UNDP Projects in Saudi and outside
[Added: 2019/12/15] [Last Updated: 2020/05/06]
CTA/PM 2020/05 Completed New structure approved, new recruitment launched History
4. Recommendation:

Recommendation 4. Re-activate the project board and project coordination group (currently called steering committee)

This project board is critically crucial in influencing the strategic directions of the project. The board is a strong governance platform to make on consensus-based management decisions for a project, provide strategic guidance and help to overcome risks. It is important to convene the project board with the same composition as described in the project document, at least, twice a year. The project board is particularly needed to oversee the implementation of the recommendation in this evaluation. 

It is also recommended that the project steering committee ToR is reviewed with an intent to make this a coordination platform only and no decision making, this may require to call it as Project Coordination Group (PCG) for technical coordination, harmonization, communication and linkages. The PCG should meet, at least, on a monthly basis to ensure proper coordination happening. 

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Suggested Groups will be reactivated and meeting plan will be drafted
[Added: 2019/12/15] [Last Updated: 2020/02/04]
PM/CTA 2020/01 Completed A plan has been drafted, Project board re-activated and new decisions to meet all other recommendations are currently being undertaken History
5. Recommendation:

Recommendation 5. UNDP to support the project with communication expertise

 It is obvious that there is inadequate communication of project achievements to MEWA decision makers and broader stakeholders. It is understood that UNDP CO has recently established a communication capacity to support various programs, and it is highly recommended that a communication specialist supports the PMU to communicate the project achievements appropriately and effectively. 

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Management and UNDP will meet to discuss

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Steering committee meeting in January will identify needs and assist project in recruitment during Q1 2020
[Added: 2019/12/16] [Last Updated: 2020/05/06]
Mayssam Tamim 2020/04 Completed Steering committee met, identified needs all addressed and new recruitment currently ongoing History
6. Recommendation:

Recommendation 6. Upgrade the software and hardware assets to meet the modelling technical requirements

The project failed to obtain the required hardware and software to undertake the highly technical modelling work, as this is essential to undertake the modelling work on ongoing basis, it is recommended to invest in such a foundational basis to deliver high quality product and upgrade the already purchased software and hardware, and if required, buy a new ones. 

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Agreement on new experts needed will be finalized and hardware required will be purchased
[Added: 2019/12/15] [Last Updated: 2020/06/11]
Project Manager 2020/05 Completed restructuring finalized and new experts are being recruited History
7. Recommendation:

Recommendation 7. Assess the project financial needs and develop a resource mobilization plan

As noted in the evaluation, the project offers relatively limited resources comparing to the scale of the activities. The project is highly dependent on external funding, in some cases, non-recurrent public funding. It is recommended to assess the funding gaps, map funding opportunities and develop resource mobilisation plan including scenarios for securing the required funding. 

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Additional personnel requirements identified Detailed budget review and changes for sustainability made
[Added: 2019/12/15] [Last Updated: 2020/06/11]
CTA/PM 2020/05 Completed restructuring finalized and new personnel is being recruited History
TOR for ”Fund Raiser” Expert ready, Plan to be finalized by expert
[Added: 2019/12/15] [Last Updated: 2020/06/11]
CTA/PM 2020/06 Completed New advisor to deputy minister hired to assist with fund raising for ministry and project History
8. Recommendation:

Recommendation 8. Review the project resourcing strategy and ToRs of all experts and address the deficit

It is recommended to undertake a holistic review of all experts ToRs to be more driven by the needs under each output, and balance the experts allocations to different outputs as well as review the expert engagement modality including short-term assignment in case if the sought deliverables don’t necessarily need a full time-job.

In light of these proposed changes in experts engagement modalities and other changes in resourcing strategy, the budget deficit need to be addressed as well.

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
All TORs of experts will be reviewed and resources availability and planning will be done
[Added: 2019/12/16] [Last Updated: 2021/04/12]
CTA/PM 2021/03 Completed All Experts TORs have indeed been revised to match IPSA contract modality History
9. Recommendation:

Recommendation 9. Design fit-for-purpose reporting templates with a greater focus on the M&E framework – including the performance indicators

The monitoring template used by the project includes a long list of questions - in addition to the reporting on performance indicators - that need to be answered to complete a semi-annual monitoring report. It is too time consuming, cumbersome to complete and the result is that it does not provide accurate and timely information on how the project is progressing. There is a need to review the reporting template, to shorten it and focus on the performance framework with the set of indicators as the central part to measure how well the project is progressing toward the achievement of its expected outputs and outcomes.

On that front, it is important to guide the project management to deliver fit-forpurpose reporting. For example, developing a 1-page dashboard reporting template targeting decision makers. 

Management Response: [Added: 2019/11/20] [Last Updated: 2020/11/14]

Recommendation taken under advisement

Key Actions:

Key Action Responsible DueDate Status Comments Documents
KPIs being developed.
[Added: 2019/12/15] [Last Updated: 2021/08/29]
CTA 2021/08 Completed Meeting done, final report and presentation submitted to deputy minister. History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org