Success message
error message
warn message
Yemen Emergency Crisis Response Project
Commissioning Unit: Yemen
Evaluation Plan: 2012-2019
Evaluation Type: Project
Completion Date: 08/2019
Unit Responsible for providing Management Response: Yemen
Documents Related to overall Management Response:
 
1. Recommendation:

SFD should use their on-the-ground experience to provide feedback through UNDP to USAID on each type of activity design implemented under this ECRP component. This advice should take the form of a set of options that may assist improving outcomes without overly compromising implementation efficiency. It should not take the form of merely presenting preferred options.

Lesson 2. The prepackaged nature of many of the activities had both positive and negative effects on implementation and outcomes.

The fact that many of the activities under this component of ECRP where largely pre-designed by the donor (USAID) allowed rapid application of these designs on the ground by UNDP and SFD, as there was little or no design-lag to factor into schedules. Given the extremely tight schedules, as well as the late stage increases in programming faced by ECRP, it is unlikely that the magnitude of work completed could have been achieved otherwise. There is no doubt that this component of ECRP got a lot of assistance on the ground quickly.

However, this benefit was not without its own costs. Fixed approaches, limited flexibility in timing of activities, and short delivery windows definitely had an effect on the quality of assistance delivered and consequently on the resulting outcomes of the work.  Training courses were often intensive, and perhaps even rushed, with associated limitations on practical exercises. Timing of assistance for fishermen and farmers could have benefited from avoiding peak fishing season or aligning with the growing season, respectively.  Greater consideration might have been given to overcoming remaining barriers to vulnerable groups accessing the improved service delivery generated by the activities (e.g. the high cost of medicines still associated with health services), or broader implications of participation costs (e.g. fishermen accruing debt to pay part costs of assistance).

As always, when faced with the pros and cons of different approaches to dealing with complex real-world contexts, it is not a question of attaining perfection. Rather, it is about achieving an optimal balance. One approach that may help identify where these optimum lies is provision of some simple and iterative feedback by the implementer. SFD is currently in an ideal position to provide advice to USAID (through UNDP), about tweaking the prepackaged designs for the activities that have just been implemented. This advice could take the form of a set of ‘options’, for each activity type that USAID may consider. This advice needs to be both well justified and frank, presenting both pros and cons of each option, rather than simply advocating for a preferred model.

Management Response: [Added: 2019/12/12] [Last Updated: 2020/11/24]

Recommendation is well taken. SFD reports to UNDP as per the agreement signed with UNDP, and UNDP in turn report to USAID as per the agreement terms. Any issues from project implementation, including risks and challenges are included in these reports.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
SFD will be requested to provide regular feedback on the practicality and applicability of each activity. That may help improve the outcomes while maintaining the quality and efficiency of implementation. This should be detailed in the project documents for future funding or in the LoAs.
[Added: 2019/12/12] [Last Updated: 2020/10/08]
SFD, and UNDP (ERR Unit) 2020/06 No Longer Applicable [Justification: Suspended. No further funding or discussion on new funds from USAID until now. However, overall improvements in quality and learning have been considered in other funds as indicated below. ]
History
2. Recommendation:

Future M&E efforts for these types of activities should include real-time outcome monitoring, based on measures derived from stated objectives. In the absence of more direct measures, systematic and representative gathering of qualitative examples of actions or behaviors that potentially satisfy these objectives should be considered, with post-collection classification of these examples used as a means of quantitatively analyzing this information.

 

Lesson 3.  Monitoring of activity outcomes need not be constrained to limited, ex-post evaluation exercises such as this one, and real time outcome monitoring will be facilitated by development of some basic outcome indicators that reflect the stated objectives of the assistance.

The objectives of this component of ECRP are appropriately modest:

(i) Women and youth have enhanced skills to be employed in education

(ii) Community mid-wives and paramedics have enhanced capacities to provide good quality private health services

 (iii) Fishermen, coffee, and crop farmers are enabled to expand their production through the development of the value chain, usage of modern inputs and improved practices

(iv) Oversight, reporting and quality assurance of project ensured

Focusing on the first three objectives (as the fourth is management focused), each of these may be relatively easily measured by robustly assuming the best way to assess if skills are acquired is to look for their use. In fact, the first two objectives could arguably be measured in terms of educational outcomes alone, however this would not reveal the associated increases in service delivery achieved.

Rather than focusing of measures of beneficiary ‘satisfaction’, as much of the current third-party monitoring approach appears to do, it may be more useful to systematically gather examples of skills use, and the classify them in relation to their pertinence to each of the first three objectives. Appropriate sampling could be used to ensure that the full set of examples gathered is representative of the overall results, and the classification of examples would allow for more quantitative analyses to be conducted. This approach would provide direct evidence of success at delivering the desired outcomes in real time, so that adjustments may be made in similar activities if necessary, prior to completion of the entire component.

Management Response: [Added: 2019/12/12] [Last Updated: 2020/11/24]

Recommendation is considered for future funding by all donors. Given the security situation in the country, UNDP engaged a third-party monitoring agency (TPMA) to undertake field-based monitoring.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
TPMA and UNDP ECRP Project is to gather a systematic and representative qualitative example in case of the absence of more direct measures.
[Added: 2019/12/12] [Last Updated: 2020/11/10]
UNDP (ERR UNIT) and TPMA. 2020/12 Completed TPM exercises include qualitative data collection approaches in their methodology History
3. Recommendation:

Recommendation 1. While giving due considerations to the recommendations that follow, if at all possible, funding should be made available for the work of this component of ECRP should be consolidated and continued.

 

Lesson 1. In a context such as Yemen, in which needs are massive, all assistance of this type is both appreciated and of significant benefit.

This lesson is included to ensure the further lessons and recommendations provided below do not overshadow the on-the-ground achievements to date of this component of ECRP. The activities under this component were not perfect, but they were successful in substantially increasing essential service delivery in the locations they targeted. This improved or renewed service delivery is likely to have strongly positive effects on stability and resilience of these locations.

Management Response: [Added: 2020/11/24] [Last Updated: 2021/06/03]

Recommendation is accepted. UNDP will continue to seek funding to finance related activities. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
continuation of ECRP activities
[Added: 2021/02/25]
UNDP ECRP team 2019/12 Completed Implementation of ECRP activities continued with new funding from KSA History
4. Recommendation:

Recommendation 4. Future agreements with implementer must include requirements for timely provision of lists of participants, including as many telephone contact details as possible. Third party monitoring should select random samples of these participants to contact and confirm details of assistance received in order to meet routine triangulation standards.

Lesson 4.  Third party monitoring of activity outputs should not be based on implementer reporting, as this still constitutes single source reporting which is incompatible with the accountability function of third-party monitoring.

While the evaluation itself triangulated some of the implementing partner’s output data and found no reason to doubt the veracity of SFD reporting, it remains the role of third party monitoring to independently confirm both the quantity and quality of claimed. This is not a simple task and is contingent upon UNDP and its third-party monitors having independent access to activity beneficiaries. As UNDP field reports noted, giving participants access to UNDP in the form of complaints mechanisms is not an equivalent function, and remains easy for implementers to gate keep. The key to third party monitoring obtaining real-time and independent access to beneficiaries is timely provision of participant lists by the implementer. Ideally, the implementer should be required to provide these lists prior to, or immediately on, activity completion, and the list should contain telephone contact details for as many participants as feasible. It is recognized that in the context of Yemen, there will be limitations in this regard, but provided a sufficient percentage of participants can be contacted and choice of sample participants is randomized, a reasonably representative sample of participants may be obtained. This sample would provide a basis for triangulation of outputs (e.g. participant numbers, activity locations, etc.). Note contact with this sample of participants need not constitute a ‘survey’. The all that needs to be asked of them is a brief description of the assistance they are receiving.  However, a proportion of those contacted may be subject to additional questions, if desired (e.g. for the purposes of reporting against outcome indicators -see Lesson 3).

Management Response: [Added: 2020/11/24] [Last Updated: 2021/06/03]

The recommendation is accepted. UNDP will ensure that all monitoring and evaluation exercises apply rigorous methodologies to enhance validity and utilisation of the results. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
TPM exercises benefitted from field data collection following rigorous methologies,including surveys, interviews with beneficiaries and stakeholders at the field level
[Added: 2021/02/25]
UNDP 2019/12 Completed TPM agencies employed rigorous methodologies in udertaking TPMs for ongoing ECRP activities

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org