- Evaluation Plan:
- 2012-2019, Yemen
- Evaluation Type:
- Final Project
- Planned End Date:
- 06/2019
- Completion Date:
- 08/2019
- Status:
- Completed
- Management Response:
- Yes
- Evaluation Budget(US $):
- 50,000
Yemen Emergency Crisis Response Project
| Document | Type | Language | Size | Status | Downloads |
|---|---|---|---|---|---|
 ToR_National Consulting Agent_ USAID_Final Evaluation FINAL 17012019.docx.pdf |
tor | English | 879.02 KB | Posted | 376 |
 Final Evaluation Report_USAID Funded SFD Components of ECRP.docx |
report | English | 1566.54 KB | Posted | 332 |
| Title | Yemen Emergency Crisis Response Project | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Atlas Project Number: | 00099947,00097850 | ||||||||||||
| Evaluation Plan: | 2012-2019, Yemen | ||||||||||||
| Evaluation Type: | Final Project | ||||||||||||
| Status: | Completed | ||||||||||||
| Completion Date: | 08/2019 | ||||||||||||
| Planned End Date: | 06/2019 | ||||||||||||
| Management Response: | Yes | ||||||||||||
| UNDP Signature Solution: |
|
||||||||||||
| Corporate Outcome and Output (UNDP Strategic Plan 2018-2021) |
|
||||||||||||
| SDG Goal |
|
||||||||||||
| SDG Target |
|
||||||||||||
| Evaluation Budget(US $): | 50,000 | ||||||||||||
| Source of Funding: | |||||||||||||
| Evaluation Expenditure(US $): | 50,000 | ||||||||||||
| Joint Programme: | No | ||||||||||||
| Joint Evaluation: | No | ||||||||||||
| Evaluation Team members: |
|
||||||||||||
| GEF Evaluation: | No | ||||||||||||
| Key Stakeholders: | World Bank | ||||||||||||
| Countries: | YEMEN |
| Lessons | |
|---|---|
| Findings | |
| 1. | Only in two cases (indicators 1.5 and 3.2.3) are results not either within a few percentage points of the targets or significantly in excess of those targets. For those indicators with applicable targets, 50% of indicators show results over 30% greater in quantity than stated targets. This is a very strong reported result. [1] Note the likely challenges in meeting gender targets at the sub-project level were recognized and declared at the proposal stage and relate to valid pragmatic constraints on drawing female candidates from severely male dominated roles or occupations. Tag: Women's Empowerment Results-Based Management Education Youth |
| 2. | In evaluative terms, the highly positive initial findings presented above have a, perhaps unwelcome, consequence. The greater the claims of success made, the more robust must be the evidence-base produced to support such claims. Given these findings are based on existing reported data, it is necessary to examine the limitations of this data to enable appropriate qualification of the initial findings. The limitations identified on the basis of desk study are: The majority of data relating to magnitude of outputs, including all data relevant to quantitative indicators and meeting of their associated targets is single-source, being derived from implementing partner reporting. This IP reporting forms the main basis of both progress reports and final summaries against agreed indicators. While this reporting framework and its application has been strong, it is limited to reporting against outputs, rather than outcomes. Tag: Impact Relevance Monitoring and Evaluation Results-Based Management |
| 3. | M&E tends to be strongly output focused, with efforts at collecting higher-level outcome data less systematized, and often opportunistic. Much of project data gathered is focused on simple quantity or quality of outputs. Systematic approaches to assessing the effects of delivering these outputs do not appear to be in place. Some recording of such outcomes does occur, but it tends to be captured in ad-hoc or opportunistic approaches, such as field visits. Third party monitoring does attempt outcome monitoring to a limited extent, but even this tends to use ‘aspirational’, rather than actual measures. Tag: Impact Monitoring and Evaluation Results-Based Management |
| 4. | The ToR for this evaluation poses a large number (36) of Evaluative Questions (EQs). The Evaluation methodology therefore needed to ‘cast a broad net’ in its data gathering. This unavoidably means some EQs are responded to more fully than others, but the methodology was generally successful in allowing all questions to be at least partially addressed. Tag: Impact Monitoring and Evaluation Results-Based Management |
| 5. | 3.2.1 Numerical Analysis of Beneficiary SurveyNote these tables represent the findings from all beneficiary surveys. They use the data from 100% of beneficiaries surveyed, and simply provide the proportion of that total who provided the responses indicated in each table. In the cases where issues raised by beneficiaries are presented, a 10% cut off is applied (i.e. only issues reported by more than 10% of beneficiaries are provided), to avoid focus on trivialities. In cases where outcomes are presented (including the sustainability of outcomes), a 1% cut off is used to ensure the fullest picture of results is provided. Tag: Impact Monitoring and Evaluation Results-Based Management Capacity Building |
| 6. | Sustainability of Benefits The beneficiary survey conducted for this Evaluation also investigated the continuation of benefits post-assistance. Respondents were asked to identify both benefits that they still enjoyed and those that were no longer being realized (at the time of the survey). Again, these responses were collected as narrative qualitative examples and then coded to allow statistical summaries. All responses that were provided by Tag: Sustainability Knowledge management Monitoring and Evaluation Results-Based Management Capacity Building Education |
| 7. | Improved or Increased Service Provision The beneficiary survey also directly investigated improved or increased service provision by beneficiaries. Again, these responses were collected as narrative qualitative examples and then coded to allow statistical summaries. All responses that were provided by more than 1% of the sample are included below. Double counting of respondents who may have raised more than one issue within a single code is excluded from this data. Tag: Impact Gender Equality Health Sector Nutrition Monitoring and Evaluation Education |
| Recommendations | |
|---|---|
| 1 | SFD should use their on-the-ground experience to provide feedback through UNDP to USAID on each type of activity design implemented under this ECRP component. This advice should take the form of a set of options that may assist improving outcomes without overly compromising implementation efficiency. It should not take the form of merely presenting preferred options. Lesson 2. The prepackaged nature of many of the activities had both positive and negative effects on implementation and outcomes. The fact that many of the activities under this component of ECRP where largely pre-designed by the donor (USAID) allowed rapid application of these designs on the ground by UNDP and SFD, as there was little or no design-lag to factor into schedules. Given the extremely tight schedules, as well as the late stage increases in programming faced by ECRP, it is unlikely that the magnitude of work completed could have been achieved otherwise. There is no doubt that this component of ECRP got a lot of assistance on the ground quickly. However, this benefit was not without its own costs. Fixed approaches, limited flexibility in timing of activities, and short delivery windows definitely had an effect on the quality of assistance delivered and consequently on the resulting outcomes of the work. Training courses were often intensive, and perhaps even rushed, with associated limitations on practical exercises. Timing of assistance for fishermen and farmers could have benefited from avoiding peak fishing season or aligning with the growing season, respectively. Greater consideration might have been given to overcoming remaining barriers to vulnerable groups accessing the improved service delivery generated by the activities (e.g. the high cost of medicines still associated with health services), or broader implications of participation costs (e.g. fishermen accruing debt to pay part costs of assistance). As always, when faced with the pros and cons of different approaches to dealing with complex real-world contexts, it is not a question of attaining perfection. Rather, it is about achieving an optimal balance. One approach that may help identify where these optimum lies is provision of some simple and iterative feedback by the implementer. SFD is currently in an ideal position to provide advice to USAID (through UNDP), about tweaking the prepackaged designs for the activities that have just been implemented. This advice could take the form of a set of ‘options’, for each activity type that USAID may consider. This advice needs to be both well justified and frank, presenting both pros and cons of each option, rather than simply advocating for a preferred model. |
| 2 | Future M&E efforts for these types of activities should include real-time outcome monitoring, based on measures derived from stated objectives. In the absence of more direct measures, systematic and representative gathering of qualitative examples of actions or behaviors that potentially satisfy these objectives should be considered, with post-collection classification of these examples used as a means of quantitatively analyzing this information.
Lesson 3. Monitoring of activity outcomes need not be constrained to limited, ex-post evaluation exercises such as this one, and real time outcome monitoring will be facilitated by development of some basic outcome indicators that reflect the stated objectives of the assistance. The objectives of this component of ECRP are appropriately modest: (i) Women and youth have enhanced skills to be employed in education (ii) Community mid-wives and paramedics have enhanced capacities to provide good quality private health services (iii) Fishermen, coffee, and crop farmers are enabled to expand their production through the development of the value chain, usage of modern inputs and improved practices (iv) Oversight, reporting and quality assurance of project ensured Focusing on the first three objectives (as the fourth is management focused), each of these may be relatively easily measured by robustly assuming the best way to assess if skills are acquired is to look for their use. In fact, the first two objectives could arguably be measured in terms of educational outcomes alone, however this would not reveal the associated increases in service delivery achieved. Rather than focusing of measures of beneficiary ‘satisfaction’, as much of the current third-party monitoring approach appears to do, it may be more useful to systematically gather examples of skills use, and the classify them in relation to their pertinence to each of the first three objectives. Appropriate sampling could be used to ensure that the full set of examples gathered is representative of the overall results, and the classification of examples would allow for more quantitative analyses to be conducted. This approach would provide direct evidence of success at delivering the desired outcomes in real time, so that adjustments may be made in similar activities if necessary, prior to completion of the entire component. |
| 3 | Recommendation 1. While giving due considerations to the recommendations that follow, if at all possible, funding should be made available for the work of this component of ECRP should be consolidated and continued.
Lesson 1. In a context such as Yemen, in which needs are massive, all assistance of this type is both appreciated and of significant benefit. This lesson is included to ensure the further lessons and recommendations provided below do not overshadow the on-the-ground achievements to date of this component of ECRP. The activities under this component were not perfect, but they were successful in substantially increasing essential service delivery in the locations they targeted. This improved or renewed service delivery is likely to have strongly positive effects on stability and resilience of these locations. |
| 4 | Recommendation 4. Future agreements with implementer must include requirements for timely provision of lists of participants, including as many telephone contact details as possible. Third party monitoring should select random samples of these participants to contact and confirm details of assistance received in order to meet routine triangulation standards. Lesson 4. Third party monitoring of activity outputs should not be based on implementer reporting, as this still constitutes single source reporting which is incompatible with the accountability function of third-party monitoring. While the evaluation itself triangulated some of the implementing partner’s output data and found no reason to doubt the veracity of SFD reporting, it remains the role of third party monitoring to independently confirm both the quantity and quality of claimed. This is not a simple task and is contingent upon UNDP and its third-party monitors having independent access to activity beneficiaries. As UNDP field reports noted, giving participants access to UNDP in the form of complaints mechanisms is not an equivalent function, and remains easy for implementers to gate keep. The key to third party monitoring obtaining real-time and independent access to beneficiaries is timely provision of participant lists by the implementer. Ideally, the implementer should be required to provide these lists prior to, or immediately on, activity completion, and the list should contain telephone contact details for as many participants as feasible. It is recognized that in the context of Yemen, there will be limitations in this regard, but provided a sufficient percentage of participants can be contacted and choice of sample participants is randomized, a reasonably representative sample of participants may be obtained. This sample would provide a basis for triangulation of outputs (e.g. participant numbers, activity locations, etc.). Note contact with this sample of participants need not constitute a ‘survey’. The all that needs to be asked of them is a brief description of the assistance they are receiving. However, a proportion of those contacted may be subject to additional questions, if desired (e.g. for the purposes of reporting against outcome indicators -see Lesson 3). |
1. Recommendation:
SFD should use their on-the-ground experience to provide feedback through UNDP to USAID on each type of activity design implemented under this ECRP component. This advice should take the form of a set of options that may assist improving outcomes without overly compromising implementation efficiency. It should not take the form of merely presenting preferred options.
Lesson 2. The prepackaged nature of many of the activities had both positive and negative effects on implementation and outcomes.
The fact that many of the activities under this component of ECRP where largely pre-designed by the donor (USAID) allowed rapid application of these designs on the ground by UNDP and SFD, as there was little or no design-lag to factor into schedules. Given the extremely tight schedules, as well as the late stage increases in programming faced by ECRP, it is unlikely that the magnitude of work completed could have been achieved otherwise. There is no doubt that this component of ECRP got a lot of assistance on the ground quickly.
However, this benefit was not without its own costs. Fixed approaches, limited flexibility in timing of activities, and short delivery windows definitely had an effect on the quality of assistance delivered and consequently on the resulting outcomes of the work. Training courses were often intensive, and perhaps even rushed, with associated limitations on practical exercises. Timing of assistance for fishermen and farmers could have benefited from avoiding peak fishing season or aligning with the growing season, respectively. Greater consideration might have been given to overcoming remaining barriers to vulnerable groups accessing the improved service delivery generated by the activities (e.g. the high cost of medicines still associated with health services), or broader implications of participation costs (e.g. fishermen accruing debt to pay part costs of assistance).
As always, when faced with the pros and cons of different approaches to dealing with complex real-world contexts, it is not a question of attaining perfection. Rather, it is about achieving an optimal balance. One approach that may help identify where these optimum lies is provision of some simple and iterative feedback by the implementer. SFD is currently in an ideal position to provide advice to USAID (through UNDP), about tweaking the prepackaged designs for the activities that have just been implemented. This advice could take the form of a set of ‘options’, for each activity type that USAID may consider. This advice needs to be both well justified and frank, presenting both pros and cons of each option, rather than simply advocating for a preferred model.
Management Response: [Added: 2019/12/12] [Last Updated: 2020/11/24]
Recommendation is well taken. SFD reports to UNDP as per the agreement signed with UNDP, and UNDP in turn report to USAID as per the agreement terms. Any issues from project implementation, including risks and challenges are included in these reports.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| SFD will be requested to provide regular feedback on the practicality and applicability of each activity. That may help improve the outcomes while maintaining the quality and efficiency of implementation. This should be detailed in the project documents for future funding or in the LoAs.
[Added: 2019/12/12] [Last Updated: 2020/10/08] |
SFD, and UNDP (ERR Unit) | 2020/06 | No Longer Applicable |
[Justification: Suspended. No further funding or discussion on new funds from USAID until now. However, overall improvements in quality and learning have been considered in other funds as indicated below. ] History |
2. Recommendation:
Future M&E efforts for these types of activities should include real-time outcome monitoring, based on measures derived from stated objectives. In the absence of more direct measures, systematic and representative gathering of qualitative examples of actions or behaviors that potentially satisfy these objectives should be considered, with post-collection classification of these examples used as a means of quantitatively analyzing this information.
Lesson 3. Monitoring of activity outcomes need not be constrained to limited, ex-post evaluation exercises such as this one, and real time outcome monitoring will be facilitated by development of some basic outcome indicators that reflect the stated objectives of the assistance.
The objectives of this component of ECRP are appropriately modest:
(i) Women and youth have enhanced skills to be employed in education
(ii) Community mid-wives and paramedics have enhanced capacities to provide good quality private health services
(iii) Fishermen, coffee, and crop farmers are enabled to expand their production through the development of the value chain, usage of modern inputs and improved practices
(iv) Oversight, reporting and quality assurance of project ensured
Focusing on the first three objectives (as the fourth is management focused), each of these may be relatively easily measured by robustly assuming the best way to assess if skills are acquired is to look for their use. In fact, the first two objectives could arguably be measured in terms of educational outcomes alone, however this would not reveal the associated increases in service delivery achieved.
Rather than focusing of measures of beneficiary ‘satisfaction’, as much of the current third-party monitoring approach appears to do, it may be more useful to systematically gather examples of skills use, and the classify them in relation to their pertinence to each of the first three objectives. Appropriate sampling could be used to ensure that the full set of examples gathered is representative of the overall results, and the classification of examples would allow for more quantitative analyses to be conducted. This approach would provide direct evidence of success at delivering the desired outcomes in real time, so that adjustments may be made in similar activities if necessary, prior to completion of the entire component.
Management Response: [Added: 2019/12/12] [Last Updated: 2020/11/24]
Recommendation is considered for future funding by all donors. Given the security situation in the country, UNDP engaged a third-party monitoring agency (TPMA) to undertake field-based monitoring.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| TPMA and UNDP ECRP Project is to gather a systematic and representative qualitative example in case of the absence of more direct measures.
[Added: 2019/12/12] [Last Updated: 2020/11/10] |
UNDP (ERR UNIT) and TPMA. | 2020/12 | Completed | TPM exercises include qualitative data collection approaches in their methodology History |
3. Recommendation:
Recommendation 1. While giving due considerations to the recommendations that follow, if at all possible, funding should be made available for the work of this component of ECRP should be consolidated and continued.
Lesson 1. In a context such as Yemen, in which needs are massive, all assistance of this type is both appreciated and of significant benefit.
This lesson is included to ensure the further lessons and recommendations provided below do not overshadow the on-the-ground achievements to date of this component of ECRP. The activities under this component were not perfect, but they were successful in substantially increasing essential service delivery in the locations they targeted. This improved or renewed service delivery is likely to have strongly positive effects on stability and resilience of these locations.
Management Response: [Added: 2020/11/24] [Last Updated: 2021/06/03]
Recommendation is accepted. UNDP will continue to seek funding to finance related activities.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| continuation of ECRP activities
[Added: 2021/02/25] |
UNDP ECRP team | 2019/12 | Completed | Implementation of ECRP activities continued with new funding from KSA History |
4. Recommendation:
Recommendation 4. Future agreements with implementer must include requirements for timely provision of lists of participants, including as many telephone contact details as possible. Third party monitoring should select random samples of these participants to contact and confirm details of assistance received in order to meet routine triangulation standards.
Lesson 4. Third party monitoring of activity outputs should not be based on implementer reporting, as this still constitutes single source reporting which is incompatible with the accountability function of third-party monitoring.
While the evaluation itself triangulated some of the implementing partner’s output data and found no reason to doubt the veracity of SFD reporting, it remains the role of third party monitoring to independently confirm both the quantity and quality of claimed. This is not a simple task and is contingent upon UNDP and its third-party monitors having independent access to activity beneficiaries. As UNDP field reports noted, giving participants access to UNDP in the form of complaints mechanisms is not an equivalent function, and remains easy for implementers to gate keep. The key to third party monitoring obtaining real-time and independent access to beneficiaries is timely provision of participant lists by the implementer. Ideally, the implementer should be required to provide these lists prior to, or immediately on, activity completion, and the list should contain telephone contact details for as many participants as feasible. It is recognized that in the context of Yemen, there will be limitations in this regard, but provided a sufficient percentage of participants can be contacted and choice of sample participants is randomized, a reasonably representative sample of participants may be obtained. This sample would provide a basis for triangulation of outputs (e.g. participant numbers, activity locations, etc.). Note contact with this sample of participants need not constitute a ‘survey’. The all that needs to be asked of them is a brief description of the assistance they are receiving. However, a proportion of those contacted may be subject to additional questions, if desired (e.g. for the purposes of reporting against outcome indicators -see Lesson 3).
Management Response: [Added: 2020/11/24] [Last Updated: 2021/06/03]
The recommendation is accepted. UNDP will ensure that all monitoring and evaluation exercises apply rigorous methodologies to enhance validity and utilisation of the results.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| TPM exercises benefitted from field data collection following rigorous methologies,including surveys, interviews with beneficiaries and stakeholders at the field level
[Added: 2021/02/25] |
UNDP | 2019/12 | Completed | TPM agencies employed rigorous methodologies in udertaking TPMs for ongoing ECRP activities |