- Evaluation Plan:
- 2012-2016, Mongolia
- Evaluation Type:
- Final Project
- Planned End Date:
- 01/2015
- Completion Date:
- 01/2015
- Status:
- Completed
- Management Response:
- Yes
- Evaluation Budget(US $):
- 20,000
Final Evaluation of the project Scaling up support to Local Service Delivery for the MDGs: Rural Water and Sanitation
| Document | Type | Language | Size | Status | Downloads |
|---|---|---|---|---|---|
 Final Evaluation Report ROK-UNDP WATSAN 5MB.pdf |
report | English | 4940.94 KB | Posted | 340 |
| RWS MR_11 June 2015.pdf |
related-document | English | 1060.30 KB | Posted | 218 |
| Title | Final Evaluation of the project Scaling up support to Local Service Delivery for the MDGs: Rural Water and Sanitation | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Atlas Project Number: | 00086612 | ||||||||
| Evaluation Plan: | 2012-2016, Mongolia | ||||||||
| Evaluation Type: | Final Project | ||||||||
| Status: | Completed | ||||||||
| Completion Date: | 01/2015 | ||||||||
| Planned End Date: | 01/2015 | ||||||||
| Management Response: | Yes | ||||||||
| Focus Area: |
|
||||||||
| Corporate Outcome and Output (UNDP Strategic Plan 2014-2017) |
|
||||||||
| Evaluation Budget(US $): | 20,000 | ||||||||
| Source of Funding: | MDG Trust Fund | ||||||||
| Joint Programme: | No | ||||||||
| Joint Evaluation: | No | ||||||||
| Evaluation Team members: |
|
||||||||
| GEF Evaluation: | No | ||||||||
| Key Stakeholders: | Ministry of Construction and Urban Development, Local Governments, Ministry of Finance | ||||||||
| Countries: | MONGOLIA |
| Lessons | |
|---|---|
| Findings |
| Recommendations | |
|---|---|
| 1 | UNDP is strongly urged to consider the benefits of undertaking an ex-post evaluation of this Project in one to two years time. UNDP and KOICA might like to consider the potential benefits of combining an expost evaluation with a scoping study for potential future support to rural water and sanitation in Mongolia, building on the results and lessons learnt from this Project. Opportunities exist for UNDP and KOICA to build on the partnerships that they have forged under this Project in order to develop and increase the impact of their work in support of water and sanitation service provision within sustainable rural development agendas. In 2016 KOICA Mongolia Office will be developing a new Country Partnership Strategy with a likely increased focus on rural development. They view UNDP as an important partner in this area. UNDP Mongolia Country Office will also be developing a new CPAP. There would be significant benefits for both organisations in close collaboration during the development of their respective strategic directions, to explore opportunities for harmonization. |
| 2 | There is a need to ensure that Project Documents clearly outline intended impacts and results; this includes the need to incorporate specific, measurable, achievable, relevant and time bound (SMART) indicators and targets, as a means of measuring project results and impact. This is essential for effective monitoring and evaluation of project results and in order to guide results based management during project implementation. The ROK-UNDP MDG Trust Fund might like to look at developing guidance to strengthen Project Design templates for UNDP Country Offices and partner organisations to ensure that Project Documents incorporate clear Result and Impact indicators, and that Project monitoring and evaluation frameworks meet international standards. |
| 3 | Where significant changes to the scale, intended impact or focus of a project are made during or following inception, corresponding amendments should be made to the Project Document, or an addendum added to the Project Document, agreed with all relevant parties, and signed off by the Project Board/Steering Committee. Risk assessment should be undertaken to assess the implications of changes to the scale, scope or approach of a Project. Impact indicators should correspondingly be amended in the project?s monitoring and evaluation framework. It would be useful for UNDP and/or the ROK-UNDP MDG Trust Fund to develop guidance in this area. |
| 4 | Adequate timeframes are required to support water and sanitation infrastructure projects of this scale in Mongolia, particularly given the short construction season. Timeframe is obviously relative to scale of infrastructural development proposed, but it is important that this is fully considered as part of Project design, including comprehensive risk assessment. |
| 5 | Donors and national/local funding agencies are advised to fully assess total funding required to establish operational WATSAN systems, prior to commissioning their installation. Following completion of, and based on, the Detailed Design Drawings for Construction (DDDC), donor and national partners will be able to assess the funds required for completion of functional WATSAN facilities. This information would strongly support risk assessment by all partners. Where a donor funded project is contributing to construction of elements of WATSAN systems, the evaluator suggests that it would be sensible for project funds to be released only once other sources have been identified to meet total funding required to complete the facilities. This approach would significantly reduce the risk of insufficiency of funds to complete WATSAN systems following the end of a project. At the five remaining Soums supported though this ROK-UNDP MDG Trust Fund Project, there is an urgent need for MCUD and local authorities to find the funds required to complete the installation of centralised WATSAN facilities, and to ensure that these become operational. |
| 6 | It is important that donors and national/local agencies assess the financial viability of operation and maintenance of WATSAN systems as part of decision making on whether to install them or not. The decision on whether to install a centralised WATSAN should include assessment of the financial viability of service delivery (based on technical data relative to the capacity of the system / number of proposed users). It will then be possible for decision makers to estimate the number of users/tariff rates required for sustainable operation, as part of feasibility analysis. At all 8 Soums supported though this ROK-UNDP MDG Trust Fund Project, there is an urgent need to assess the predicted costs of service delivery to the public service buildings that are/will be connected to the centralised WATSAN systems, and to assess financial implications of increasing the number of residents connected into the system. In relation to this it will be important to assess the cash flow and liquidity of PUSO responsible for local service delivery at each site, and the implications of this for revenue generation through water tariffs. This will enable local and national authorities, and PUSO, to assess the financial sustainability of the tripartite model at each site, and to identify mechanisms to ensure sustainable service provision at each site. |
1. Recommendation: UNDP is strongly urged to consider the benefits of undertaking an ex-post evaluation of this Project in one to two years time. UNDP and KOICA might like to consider the potential benefits of combining an expost evaluation with a scoping study for potential future support to rural water and sanitation in Mongolia, building on the results and lessons learnt from this Project.
Opportunities exist for UNDP and KOICA to build on the partnerships that they have forged under this Project in order to develop and increase the impact of their work in support of water and sanitation service provision within sustainable rural development agendas. In 2016 KOICA Mongolia Office will be developing a new Country Partnership Strategy with a likely increased focus on rural development. They view UNDP as an important partner in this area. UNDP Mongolia Country Office will also be developing a new CPAP. There would be significant benefits for both organisations in close collaboration during the development of their respective strategic directions, to explore opportunities for harmonization.
Management Response: [Added: 2016/01/13]
It agrees to the importance of an ex-post evaluation/review of the project in one or two years? time. The evaluation will be covered as part of the Outcome evaluation for the environment portfolio, which is planned in the Q4, 2015. Results and lessons learnt from this project as well as the findings of the outcome evaluation will inform the future programme/project formulation on the relevant subject matter. Building on the partnerships forged under this project, UNDP will seek further opportunities to support the national water and sanitation provision services. UNDP appreciates that KOICA Mongolia Office considering us as an important partner in this area. In developing a new CPAP, internal discussions will be held on the strategic directions and harmonization of the activities.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 1.1. Outcome Evaluation is commissioned for the Environment Portfolio
a. Drafting of TOR, which includes the evaluation of the project?s impact and public announcement
b. Conduct of the evaluation (independent team)
[Added: 2016/01/13] |
TL and CO M&E officer | 2015/12 | Completed | ||
| 1.2 Strategic direction discussion is included in the SP alignment workshop.
[Added: 2016/01/13] |
TL | 2015/12 | Completed | ||
| 1.3 CPD/CPAP discussions and consultations are held to cover strategic directions and harmonization of the activities in the relevant area.
[Added: 2016/01/13] [Last Updated: 2016/12/02] |
Environment team | 2016/03 | Completed | New CDP approved covering relevant area. History |
2. Recommendation: There is a need to ensure that Project Documents clearly outline intended impacts and results; this includes the need to incorporate specific, measurable, achievable, relevant and time bound (SMART) indicators and targets, as a means of measuring project results and impact. This is essential for effective monitoring and evaluation of project results and in order to guide results based management during project implementation. The ROK-UNDP MDG Trust Fund might like to look at developing guidance to strengthen Project Design templates for UNDP Country Offices and partner organisations to ensure that Project Documents incorporate clear Result and Impact indicators, and that Project monitoring and evaluation frameworks meet international standards.
Management Response: [Added: 2016/01/13]
CO agrees to that there is a need to ensure clearer formulation of intended impacts and results in the project documents. New project documents approvals (as per the swimlane process) require multi-step checks and clearance by the relevant officers and pre-LPAC discussion reviews the project proposals. It will strengthen its system to measure results, incorporating SMART indicators and targets in the project documents and its results framework within this process and in the pre-LPACs.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 2.1. Rigorous check of the projects results framework during the pre-LPAC processes
[Added: 2016/01/13] |
M&E officer | 2015/12 | Completed |
3. Recommendation: Where significant changes to the scale, intended impact or focus of a project are made during or following inception, corresponding amendments should be made to the Project Document, or an addendum added to the Project Document, agreed with all relevant parties, and signed off by the Project Board/Steering Committee. Risk assessment should be undertaken to assess the implications of changes to the scale, scope or approach of a Project. Impact indicators should correspondingly be amended in the project?s monitoring and evaluation framework. It would be useful for UNDP and/or the ROK-UNDP MDG Trust Fund to develop guidance in this area.
Management Response: [Added: 2016/01/13]
As per the standard business procedures, changes in the project scale and focus as well as activities are discussed at the project board. CO will strengthen its system to ensure the changes made during the inception period incorporated into the Project documents as addendum and signed-off by the project boards. Risk assessments and management will be discussed rigorously at the Project Board level. Guidance will be released on both cases.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 3.1 Guidance on incorporating changes in the project document as addendum to be released.
[Added: 2016/01/13] |
M&E officer | 2015/06 | Completed | ||
| 3.2 Guidance/Interoffice memorandum on the risk management at the Project Board level to be released
[Added: 2016/01/13] |
M&E officer | 2015/05 | Completed |
4. Recommendation: Adequate timeframes are required to support water and sanitation infrastructure projects of this scale in Mongolia, particularly given the short construction season. Timeframe is obviously relative to scale of infrastructural development proposed, but it is important that this is fully considered as part of Project design, including comprehensive risk assessment.
Management Response: [Added: 2016/01/13]
A short construction window for Mongolia shall be considered as a major risk and issue for any infrastructure project. Mitigation actions should include allocation of adequate timeframe and planning of the activities procyclical to the seasonal features. In this terms, CO will pay increased attention on the planning and design of its future programme/project formulation. There is not specific action planned for this as it will depend on the funding and the strategic direction of the new CPD/CPAP.
Key Actions:
5. Recommendation: Donors and national/local funding agencies are advised to fully assess total funding required to establish operational WATSAN systems, prior to commissioning their installation. Following completion of, and based on, the Detailed Design Drawings for Construction (DDDC), donor and national partners will be able to assess the funds required for completion of functional WATSAN facilities. This information would strongly support risk assessment by all partners.
Where a donor funded project is contributing to construction of elements of WATSAN systems, the evaluator suggests that it would be sensible for project funds to be released only once other sources have been identified to meet total funding required to complete the facilities. This approach would significantly reduce the risk of insufficiency of funds to complete WATSAN systems following the end of a project. At the five remaining Soums supported though this ROK-UNDP MDG Trust Fund Project, there is an urgent need for MCUD and local authorities to find the funds required to complete the installation of centralised WATSAN facilities, and to ensure that these become operational.
Management Response: [Added: 2016/01/13]
It is important to assess the total funding required for any infrastructure installation especially that covers nationwide facilities including the water and sanitation provisions, which should be incorporated in the sectoral master plan. The Government committed itself for the construction of the social service infrastructures in its nationwide initiative for New Soum Development. Soum specific investment needs were only determined through detailed DDDC and therefore underestimated to some extent. On the other hand, The Government of Mongolia/Ministry of Construction and Urban Development has officially demonstrated its commitment to fund the construction of the integrated water and sanitation system in all the 8 target soums in the spring of 2015 where the UNDP?s project completed its installation part.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 5.1 Funding is secured to complete the construction of the integrated WatSan system, to which UNDP?s project has contributed (completed its part of installation).
[Added: 2016/01/13] |
Environment team | 2015/04 | Completed |
6. Recommendation: It is important that donors and national/local agencies assess the financial viability of operation and maintenance of WATSAN systems as part of decision making on whether to install them or not. The decision on whether to install a centralised WATSAN should include assessment of the financial viability of service delivery (based on technical data relative to the capacity of the system / number of proposed users). It will then be possible for decision makers to estimate the number of users/tariff rates required for sustainable operation, as part of feasibility analysis.
At all 8 Soums supported though this ROK-UNDP MDG Trust Fund Project, there is an urgent need to assess the predicted costs of service delivery to the public service buildings that are/will be connected to the centralised WATSAN systems, and to assess financial implications of increasing the number of residents connected into the system. In relation to this it will be important to assess the cash flow and liquidity of PUSO responsible for local service delivery at each site, and the implications of this for revenue generation through water tariffs. This will enable local and national authorities, and PUSO, to assess the financial sustainability of the tripartite model at each site, and to identify mechanisms to ensure sustainable service provision at each site.
Management Response: [Added: 2016/01/13]
Importance of financial assessment of the national Watsan system is acknowledged. At the same time, it acknowledges the challenges for GoM to provide its basic services to all its population. CO will review the issue as part of Outcome evaluation towards the end of 2015 on the financial viability and sustainability of the established system and management mechanism at the local level. The lessons learnt will be incorporated in the next CPAP review.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| 6.1 Lessons learnt from the project will be incorporated and discussed as part of the annual CPAP review.
[Added: 2016/01/13] [Last Updated: 2016/12/02] |
Environment team | 2016/01 | Completed | Annual CPAP review and the validation workshop of CPD conducted on February, 2016. Lessons learnt discussed. History |