Success message
error message
warn message
SUPPORT TO THE CUBANGO-OKAVANGO RIVER BASIN STRATEGIC ACTION PROGRAMME (SAP) IMPLEMENTATION - MIDTERM REVIEW
Commissioning Unit: Botswana
Evaluation Plan: 2017-2021
Evaluation Type: Project
Completion Date: 02/2021
Unit Responsible for providing Management Response: Botswana
Documents Related to overall Management Response:
 
1. Recommendation:

Mid Term Review Recommendation 1 - Improve project management for results

As exposed in the above analysis, one of the main weaknesses of the project document stands in the proposed Results Framework (RF). As it stands the RF does not allow monitoring for results as per results-based management good practice. The RF should be adjusted to adopt a full set of SMART outcome level indicators to be monitored and reported on, and better capture outcome level results.

 

The inception period was used to refine baseline information and targets but did not modify the indicators set in the Prodoc RF. It is generally good practice to review the RF as set in the Prodoc at project start in order to check the links between outcomes and outputs, and ensure indicators set at outcome level are SMART, with well-established baselines.

A draft example of an adjusted RF is proposed in this report, but it should be completed using a participatory process. Given that the SAP M&E framework is now available, and considering the fact that the project is strongly aligned with SAP results, the adjusted RF would need to be aligned with the SAP M&E framework, using the same indicators as far as possible (a few project specific indicators may still be necessary however to capture some of the project expected results however). The adjusted RF should also include gender-focused indicators, in order to better capture gender mainstreaming results of the project.

 

The adjusted RF would allow better reporting for results, in particular in PIRs, being less descriptive and based on activities conducted, and more results focused.

 

Management Response: [Added: 2021/03/15]

Recommendation Partially accepted. The project identified this issue during the inception phase, whereby some amendments and changes were made. The project has 22 indicators, which are measured through 62 targets as per the Project RF. The identified weakness may only be applicable for some of the indicators under project component 1 (Basin Development and Management Framework - BDMF), which the evaluator was able to access progress made to date varying from Moderately Satisfactory to Highly Satisfactory. However, we recognize that with the Strategic Action Programme (SAP) monitoring and evaluation framework approval in 2019, and considering that there is only a year left of actual project implementation, the ideal approach to be adopted will not be towards adjusting the Project Results Framework, rather align key output indicators (targets) with the SAP M&E framework (for those that applicable and budgeted for in 2021).

Therefore, the Project Management Unit (PMU) will work closely with OKACOM Secretariat to identify and capture from the SAP M&E Framework targets that will be aligned to the project 2021 annual targets and report on these appropriately in the upcoming PIR’s (2020/2021 and if possible 2021/2022 if still be required after project closure).

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Align project annual targets with SAP M&E Framework
[Added: 2021/03/16]
1.1 Align project annual targets with SAP M&E Framework 2021/05 Initiated Internal meeting to review SAP M&E Framework and extract relevant targets to inform 2021 AWP targets
1.2 PIR reporting on results informed from ProDoc/AWP results framework and SAP M&E framework
[Added: 2021/03/16]
1.2 PIR reporting on results informed from ProDoc/AWP results framework and SAP M&E framework 2021/07 Initiated Quality assurance of the PIR inputs from the PMU from M&E focal person to ensure that results focused reporting principles is applied and linked to set amended annual targets (informed from both ProDoc RF and SAP M&E Framework)
2. Recommendation:

Mid Term Review Recommendation 2 - Identify major risks to the project and provide clear mitigation measures and management response into a risk log

Results-based management good practice requires that risks to the project be clearly identified and mitigated through adapted and agreed management responses. This is currently lacking in the project: the Prodoc and GEF CEO Endorsement Request did not provide a real analysis of the risks to the project, of possible mitigation measures, and on the way those would be monitored and managed. PIRs reports are also deficient in terms of risk reporting.

Therefore, there is a need to identify main risks to the project, for different risk categories (political, economical, social, environmental), and propose mitigation measures, and report on those regularly

Management Response: [Added: 2021/03/15]

Management Response:

Recommendation partially accepted. Most of project risks were identified during project design phase and included in the project results framework (see ProDoc pg 34 – 46 under risks and assumptions). Similar exercise was done for each individual demonstration project. However, the risks categories are the ones missing, which the PMU with assistance from UNDP M&E focal person will act on this to categorize the identified risks and those that emerged during the project implementation, such as COVID-19 pandemic global outbreak duly captured in ATLAS risk log.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Categorize project risks (political, economic, social, environmental) and load into risk log
[Added: 2021/03/16]
- PMU - UNDP M&E focal person 2021/05 Initiated PMU will work internally to categorize the identified risks (on the ProDoc) and collect other potential risks. With support from M&E focal person and OKACOM Secretariat, risk management actions will be development and included into a risk log.
2.2 Regular monitoring of project risk and update risk log
[Added: 2021/03/16]
- PMU - UNDP M&E focal person 2021/04 Overdue-Initiated With support from M&E focal person and OKACOM Secretariat, risk management actions will be development and included into a risk log
3. Recommendation:

Mid Term Review Recommendation 3 - Build on and learn from the experience gained in demonstration projects

The project team is putting strong efforts in delivering substantial results in the demonstration projects. It is not easy, as there are many challenges in working with local communities in three different countries, through different delivery partners, and sometimes in very remote conditions. Whether those demonstration projects will deliver outstanding results or not is not, however, the most important in this project. What is key is for the project to carry on building on the experiences gained and to provide opportunities for ongoing OKACOM added value stemming from these experiences. In this sense, a replication strategy is planned during year 4 of the project.

It is therefore recommended to conduct an in-depth analysis of the success and failure factors of the demonstration projects, informing on the main challenges met, the solutions explored and the key parameters to consider when replicating the demos.

The three national governments, together with private and NGO implementing partners, have an important role to play in pursuing activities in the demos, ensuring their sustainability and long-term success. They also have responsibility in replicating the demos where suitable and relevant, and this is why a critical analysis of the demonstration projects will be useful. Such analysis will require to spend some time in the field, in every communities involved, and try to understand in each context the reasons for success or failure, so that implementation guidelines for future interventions can be prepared.

Linked to the above, a sub recommendation relates to the involvement of beneficiary communities in defining the interventions of the demonstration projects. Consultations did occur in this project, but frustrations relating to communities involvement into the design of the facilities to be constructed, or the equipment to be bought, were expressed. For the sake of effectiveness, efficiency and sustainability of the demos, it is recommended to pay specific attention to the actual involvement of beneficiary communities at all steps of the project, from initial conception to the design of specifications for construction works, implementation and delivery. Employment of local workers in construction works may also need to be more systematic or more controlled. It is therefore recommended to ensure that the level of involvement of beneficiary communities in defining the interventions of the demonstration projects is appropriate and accepted by all.

Management Response: [Added: 2021/03/15]

Recommendation fully accepted. This recommendation remains relevant for the project and the team agrees that a comprehensive replication strategy should be developed informed by lessons learned from the individual demonstration projects. As clearly indicated in the recommendation and from the project document, the replication strategy is planned for year 4 of the project, which is the current year of 2021. The project budgeted for the replication strategy development, whereby a regional / international expert will be recruited to support the PMU and the Secretariat to capture lessons learned from all demonstration projects. This will further be translated into an in-depth replication strategy informed by lessons from the 5 demonstrations and used as guiding instrument for the Commission for future low-impact socio-economic development initiatives across the basin.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Develop ToRs to recruit an IC to conduct demonstration projects replication strategy
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/02 Completed Replication strategy was planned for 2021, which ToRs are finalized to be advertised in the three riparian countries
3.2 Advert of consultancy for recruitment of the consultant
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/03 Completed
3.3 Contract awarding of IC
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/04 Overdue-Not Initiated
3.4 Field work (data collection and consultations with beneficiaries / relevant stakeholders)
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/07 Not Initiated
3.5 Data analysis and report writing
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/08 Not Initiated
3.6 Final report (lessons learned and replication strategy) submission and approval processes
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/10 Not Initiated
4. Recommendation:

Mid Term Review Recommendation 4 - Identify strategies to address shortfall of time for project delivery

To compensate the important delays occurred at project start, and then because of COVID19 pandemic, the current pace of delivery is set to complete project activities as per the project completion date of April 2022. This timing seems very tight and specific strategies should be identified to ensure the project delivers on time, engaging with national and local stakeholders a well-defined project exit strategy, and ensuring sustainability of project results.

As such, OKACOM could consider hiring more staff for specific tasks, involving delivery partners more intensively in the field, subcontracting some activities and, if necessary, reducing some activities with lower impact on final outcomes (i.e. concentrate on most impactful activities).

Management Response: [Added: 2021/03/15]

Management Response:

Recommendation fully accepted. It is important to indicate that actual project implementation timeline is up to December 2021, with the period of January to April 2022 reserved for operational closure processes. This will entail finalization of Project Terminal Evaluation (TE), 2021 FY NIM Audit, final CDR, Assets verification and transfer (if applicable), etc.

OKACOM initiated the process of seeking additional support to effectively deliver of the planned activities for 2021. This is being materialized through contracting of experts for specific deliverables to augment the existing institutional capacity at Commission level (all its structures). There are already ongoing studies through consultancy modalities on Groundwater Assessment, Sediment Assessment, Payment for Ecosystem Services and Strategic Environmental Assessment (which includes development of a transboundary Environmental Impact Assessment guidelines, socio-economic baselines, and water demand forecast).

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Contract an IC to develop lessons learned and replication strategy for the livelihood’s demonstration projects
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/10 Initiated
4.2 Procure an IC to define the Development Space concept for the CORB
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/07 Initiated Being implemented jointly with CRIDF
4.3 Engage an expert to assist the Commission in developing CORB Water Quality Standards (informed from the 4 joint surveys that) – expected to complement the Environmental Monitoring Framework
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/07 Not Initiated
4.4 Procure an IC to develop a basin IWRM Plan informed by ongoing studies on Groundwater, Sediment, SEA to further inform the conjunctive water utilization in the basin
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/09 Not Initiated This work will rely mainly from the outputs of the ongoing studies, as well as the draft EMF
5. Recommendation:

Mid Term Review Recommendation 5 - Ensure that climate change is duly considered and mainstreamed in project studies and assessments conducted, and complete a climate change sensitive IWRM plan for the basin

Climate change is a major risk to project results, and more widely in the socio-economical development and ecological balance of the CORB. In current scenarios, it is likely that the region will face more frequent and more extreme drought events, hotter conditions, and floods in the future. The latest drought events show that this has started already. It is therefore of utmost importance that all studies and assessments conducted during the project duly base their analyses on different, up-to-date climate change scenarios. This is specifically important for studies relating to water demand and water allocation, ground water assessment, sedimentation assessment, biological monitoring and socio-economic monitoring programmes. The development of an IWRM plan for the basin would constitute a major result of the project, as long as it builds up-to-date climate change scenarios, and elaborates water management options along those scenarios.

Management Response: [Added: 2021/03/15]

Management Response:

Recommendation fully accepted. This recommendation is relevant for the project and the team agrees that climate change scenarios are critical of all ongoing studies and assessment. It is important to indicate that the Commission conducted Climate Vulnerability Assessment, which identified key hotspots as priority areas for livelihoods and socio-economic development initiatives. The current studies are also being informed by this study and further refined to specific scenarios on the specialized topics, such as groundwater, sediments, biological and ecological, and socio-economic.

Most of these studies are also expected to deliver as outputs conceptual models that will further enhance the capabilities of the Decision Support System (DSS) currently under development, which will also be able to provide scenarios-based products (climate parameters being key variables) to inform the decision-making process.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Work closely with ongoing consultancy teams to ensure effective incorporation of climate change scenarios into their key deliverables
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/07 Initiated
5.2 Basin IWRM Plan reflecting climate change scenarios
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/09 Not Initiated
6. Recommendation:

Mid Term Review Recommendation 6 - Build on experience gained during this MTR when no travels are allowed

Conducting an MTR exercise without face-to-face interviews nor field visits is a challenge. Existing IT tools enable a lot, but do not replace completely real meetings and visits, which are also key moments for the consultants to more deeply understand the context in which a project is implemented.

Experience shows that conducting interviews with stakeholders in capital cities via teleconference is rather easily manageable, although sometimes impacted by connection problems or difficulties to reach people, set appointments, and more generally to get people involved.

Experience of remote field visits is interesting. The lesson learned is that for this to work really well, two main conditions must be met:

  • One person should do the field visit anyways; ideally, a local consultant should be hired and travel to the project sites to meet with local stakeholders, take pictures and mini films, and ensure the link between the evaluation team and stakeholders. In such case, the international consultant is connected “in live” to the national consultant, so they can conduct interviews together, via a 3G connection. With no national consultant or dedicated person in the field for this, conducting interviews and virtual field visits is very trick for the international consultant.
  • 3G Internet connection should be available. In very remote locations, this is a real problem and, in such cases, the national consultant needs to be able to conduct the work without the international consultant. Access to 3G connection can be set as one of the selection criteria for project site visits.
Management Response: [Added: 2021/03/15]

Recommendation fully accepted. This recommendation is relevant for the project at this time of the pandemic and the team agrees with the recommendation. Most of the project activities, even prior MTR are being conducted remotely. Our consultancies we are encouraging the bidders to have in-country team members to facilitate in data gathering and stakeholders’ consultations. Quality assurance processes are held remotely through ZOOM or MS Teams platform when it comes for specialized TRG members to provide comments and recommendations to the consultancy’s deliverables.

The project has also facilitated ordinary and extra-ordinary meetings of the Commission. 2019 was the first year in 25 years of the Commission existence that OKACOM Week (ordinary meetings of the OBSC and Council of Commission) were held remotely through zoom platform with simultaneous interpretation (English <-> Portuguese).

The project will ensure that local partners are support with necessary conditions to allow an easy interaction through various communication platforms on connection / connectivity conditions allows.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 Engage National Consultants to ensure in-country data/info is collected adequately in consultancies work across the basin to overcome travel restriction challenges imposed by the COVID19 Pandemic
[Added: 2021/03/16]
- PMU / OKACOM Secretariat 2021/10 Initiated All ongoing consultancies are following this principle, by effectively appointing in their team members representatives/consultants in each of the riparian countries

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org