MID-TERM REVIEW OF UNITED NATIONS DEVELOPMENT ASSISTANCE FRAMEWORK ETHIOPIA (UNDAF 2016-2020)

Report Cover Image
Evaluation Plan:
2017-2020, Ethiopia
Evaluation Type:
UNDAF
Planned End Date:
11/2018
Completion Date:
11/2018
Status:
Completed
Management Response:
No
Evaluation Budget(US $):
39,900

Share

Document Type Language Size Status Downloads
Download document UNDAF MTR TOR .docx tor English 73.95 KB Posted 283
Download document Ethiopia UNDAF MTR Final Report Nov 2018.pdf report English 3060.62 KB Posted 561
Title MID-TERM REVIEW OF UNITED NATIONS DEVELOPMENT ASSISTANCE FRAMEWORK ETHIOPIA (UNDAF 2016-2020)
Atlas Project Number:
Evaluation Plan: 2017-2020, Ethiopia
Evaluation Type: UNDAF
Status: Completed
Completion Date: 11/2018
Planned End Date: 11/2018
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
Evaluation Budget(US $): 39,900
Source of Funding: DOCO
Evaluation Expenditure(US $): 39,900
Joint Programme: Yes
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Richard M Chiewara International Consultant
Agaje Tesfaye Local Consultant
GEF Evaluation: No
Key Stakeholders:
Countries: ETHIOPIA
Lessons
1.

Lesson 1. To become the effective One Programme for the UN, there should be commitment and ownership of the UNDAF at all levels


2.

Lesson 2. Demonstrating the UN’s value-added requires its collective performance to be measured through specific indicators


3.

Lesson 3. Integrated joint programming requires the UNDAF to be strategically focused with an explicit theory of change


4.

Lesson 4. Programme staff fail to see the UNDAF’s added value if it does not support and enhance joint resource mobilisation


5.

Lesson 5. UN interventions can have more lasting impact by enhancing the humanitariandevelopment nexus


Findings
1.

IV. FINDINGS OF THE MID-TERM REVIEW

This chapter provides the authors’ findings based on the evaluation questions as articulated in the evaluation TORs. The findings are structured around the evaluation criteria, firstly to establish the link between each finding and the evaluation criteria, and secondly to enhance the readability of the report. Overall, the findings show mixed performance, with respect to expected results based on established benchmarks, including planned targets as well as common practices as articulated in various UN guidelines. A summary of the key findings is provided at the end of each respective section.

4.1. Relevance of the UNDAF and Delivering as One

Assessment of relevance is about establishing the degree to which the programme is related or useful in relation to a pre-defined standard or benchmark. In this connection therefore, the evaluators’ looked at the relevance of the UNDAF in relation to (a) the national development priorities, (b) Agenda 2030 for sustainable development, (SDGs), and (c) the needs of the country in light of the emerging and changing global and country context. The findings are informed by a combination of secondary data and key informant interviews. 

Finding 1: The UNDAF is aligned to national development priorities and SDGs

As noted on pages 10 and 12 above, the government’s GTP-II framework has nine pillars, while the UNDAF has five. Based on a review of the UNDAF, “it responds to eight of the nine GTP pillars showing an alignment rate of 89 per cent. This means that the UNDAF responds to all except the GTP pillar related to “building the capacity of the domestic construction industry, bridge critical infrastructure gaps with particular focus on ensuring quality provision of infrastructure services. All 15 UNDAF outcomes are directly aligned to the GTP-II priorities (Table 3), while also the indicators were taken directly from the GTP. Alignment to national priorities is good and is also a requirement for the UNDAF. However, it is also important to note that alignment does not mean doing everything. The UNDAF guidelines clearly state that the UN should identify its comparative advantages, i.e. the specific strengths that members of the UNCT bring individually and collectively in relation to other partners. Since the UN may not have adequate resources compared to other bilateral and multi-lateral partners, the identification of comparative advantages enable the UN to focus in those areas where it can make a difference. Table 3: UNDAF alignment to GTP-II and SDGs

Majority of key informants within the UN system, however felt that the UNDAF outcomes above resemble a listing of what UN agencies do according to their mandates. They noted that, while the outcomes are very much aligned to the GTP, they hardly represent the collective outcomes of the UN working together, nor do they reflect the UN’s collective comparative advantages. Majority of programme staff also said that the structure of the UNDAF only served to compartmentalise them into silos and was not conducive for joint programming and implementation. Furthermore, programme staff noted that due to the many number of outcomes/result areas, they were overstretched, and some of them had to participate in more than two result areas. In one particular case, one staff member said he was participating in 5 Results Groups. There is potential to collapse the 15 outcomes into fewer outcomes if the five UNDAF pillars were formulated as the outcomes. In order to achieve this for the next UNDAF, the UN should undertake a comprehensive country analysis as well as analysis of the UN’s collective comparative advantages. According to the 2017. UNDAF Guidelines, comparative advantage analysis ‘…is not necessarily based on those activities with which the UN system is most familiar and comfortable, focusing instead on those where the UN system can best add value’ (page 22).


Tag: Rule of law Joint UN Programme Policies & Procedures Programme Synergy Strategic Positioning UN Country Team Education Jobs and Livelihoods Leaving no one behind SDG Integration Relevance Civic Engagement Human rights

2.

4.1. Relevance of the UNDAF and Delivering as One (continuation)

Finding 3. The UNDAF lacks the strategic focus required to make it a framework for integrated programming

As noted above, the UN contributes to 15 outcomes from which it would not be farfetched to conclude that they were largely informed by a desire by UN agencies that their respective mandates should be directly included and reflected therein. 

The importance of ‘strategic focus’ is that it enables the UN to limit its activities to areas where it can pursue its core values and has a distinct comparative advantage. In order to be able to do this, the UNDAF should be based on a specific ‘theory of change’ that describes the developmental pathway to change. A word search for ‘theory of change’ in the UNDAF 2016-2020 does not yield any hits, indicating that it does not have a distinct theory of change model other than the results framework. According to 2017 UNDAF guidelines:

“UNDAFs are founded on a clearly articulated, evidence-based theory of change that describes everything that needs to happen for development change to occur. As such, the theory of change allows the UNCT to understand the ways in which the results of the UNDAF results framework relate to one another. It explains the causal relationship between different types and levels of results, and makes explicit both the risks and assumptions that define the relationship. By doing so, it allows the UNCT and its partners to interrogate those assumptions and risks when subsequently developing programmes and projects” (page 25).

The linkages between the UNDAF and SDGs are not very explicit, in part due to its lack of theory of change. A cursory review of the UNDAF shows that it purports to contribute and align with 15 SDGs (all except SDG 12 and 14). However, the UNDAF outcomes address the SDGs within their own respective silos, rather than complementing each other in a coherent manner. The 2017 UNDAF guidelines assert that “…to effectively support national efforts to achieve the transformative ambitions of the 2030 Agenda, the UN system needs to take an integrated approach to programming that combines actions across sectors and involves all relevant stakeholders. This recognizes links among the SDGs and their normative foundations”. According to one senior UN official “…what is lacking here is a UN collective vision for the country. This UNDAF tells you what we do in our agencies, but it does not tell you why we are doing it.” Clearly, there is an opportunity for the UN to rethink its contribution to the SDGs, by first interrogating what the government is doing, and articulating how the UN can apply the Mainstreaming, Acceleration and Policy Support (MAPS) approach to support the country to achieve the SDGs in a systematic, evidence-informed and results-focused way (2017 UNDAF Guidelines, p 6). 


Tag: Coherence Relevance Integration Joint UN Programme Programme Synergy Results-Based Management Theory of Change Humanitarian development nexus Peace Building Mainstreaming, Acceleration and Policy Support (MAPS) SDG Integration

3.

4.2. Effectiveness of UNDAF Implementation and Results Achievement

UNDAF effectiveness and progress towards outcome and output indicators was based on the annual reviews undertaken by Results Groups independently of the evaluation. Their assessments were further corroborated in FGDs with members of the Results Groups. It is noteworthy at this juncture to note that the FGDs were not all inclusive as some of the participating UN agencies did not attend, and in the case of Results Group 1 (Agriculture) and 6 (Health), they did not show up for the FGD. For purposes of clarity and consistence, the section is divided into two parts; the first covers general findings, and the second is more specific to respective Results Groups.

Finding 5: UNDAF outcomes and outputs reflect the UN’s core values and principlesies

The UN should reflect its core programming principles grounded in the norms and standards that it is tasked to uphold and promote – leave no one behind; human rights, gender equality and women's empowerment; sustainability and resilience; and accountability. The UNDAF outcomes and outputs reflect these core principles. According to UNDAF guidelines, ‘…as the overarching programming principle for UNDAFs in all country contexts, leaving no one behind requires that the UN system prioritize its programmatic interventions to address the situation of those most marginalized, discriminated against and excluded, and to empower them as active agents of development’ (page 9). Nineteen out of 63 outputs (30%) specifically mention and target vulnerable groups, while the others mention specifically the other programming principles. Particularly notable, is the fact that for each outcome, there is at least one output that is focusing on building capacity of national institutions, thereby ensuring both sustainability and accountability of the UN results to national priorities. In addition, there are specific outcomes on gender equality and women’s empowerment as well as human rights under the governance pillar. In this connection, it is noteworthy that programme staff observed that there were significant changes in the country’s landscape which provide increased opportunities to engage more comprehensively on human rights. For example, the government had relaxed some of the restrictive policies and legislation on civil society organisations (CSOs), thus opening up space for more engagement. With regards to sustainability and resilience, nine of the 15 outcomes have reference to sustainable and resilient products. 

Finding 6: Joint work plans are a collection of individual UN agencies’ activities and do not enhance joint implementation, monitoring or reporting

The UNDAF Results Groups (RGs) were involved in the formulation of the UNDAF and are also responsible, according to their TORs, for joint planning culminating in the joint Annual Work Plan (AWP). In that regard, the RGs developed a biennial joint work plan for 2016 and 2017. As a self-starter Delivering as One (DaO), Ethiopia is guided by the Standard Operating Procedures (SOPs) for DaO (Figure 1). However, according to RG members interviewed in FGDs, the joint planning process was mostly done independently followed by one or two meetings to compile and consolidate the work plans. Consequently, the work plans have over a thousand activities listed, which just illustrates the absence of joint planning. In fact, most of the RG lead agency/convenors noted that UN agencies rarely showed up to these meetings, and they had to resort to bilateral meetings to incorporate inputs from respective UN Agencies, Funds and Programmes (AFPs).Figure 1: Programming Framework for Delivering as One.


Tag: Effectiveness Joint UN Programme Monitoring and Evaluation Policies & Procedures Programme Synergy Project and Programme management Results-Based Management UN Agencies Coordination

4.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

Finding 7: The indicator framework does not support effective reporting of progress

The UNDAF’s indicator framework has a number of inherent weaknesses that constrains effective reporting by the RGs. This is even more apparent because the UN agencies generally implement their interventions independently and only submit reports to the RG secretariat/lead agency for compilation into the consolidated report. 

(a) Inconsistent measures. There are some inconsistences with respect to the unit of measurement for some of the indicators. The following examples are illustrative but not exhaustive.Table 5: Illustrative indicator inconsistences.

The above examples illustrate some of the challenges faced by RGs for reporting. Consequently, they use narrative information in the results matrix, because of the vagueness of the indicators.  

(b) Ambitious targets. Many of the outcome and output indicators have national-level targets taken directly from the GTP. While it is clear that the UN contributes to outcome-level results, it is still prudent that the indicator targets are realistic and attributable to the UN; otherwise the causal link between its outputs and outcomes would not be apparent. For output-level indicators however, the UN is fully accountable for their achievement, and therefore these need to be realistic, achievable and consistent with the amount of resources that will be invested. The target for indicator 6.6.3 for example is to increase the proportion of health facilities accessing safe blood supply to 100%. While this may be realistic, its achievement would be out of the UN’s control due to other factors. 

(c) Lack of data. Some of the indicators are impractical to measure. One of the problems was that the baseline data and targets was never established for some of the indicators. A cursory review shows that 15 indicators either have no baseline data or target. While it is not uncommon that some of the data may not have been available at the time of planning, it is important that the UNDAF should be updated when such data becomes available, particularly during the annual review process. In the event such data continues to be unavailable up until the mid-term review, then the indicator must either be removed or revised. The other problem with respect to data is that it is not practical to get that kind of data without a dedicated survey. For example, indicator 5.1: tons of carbon dioxide equivalent reduced. Generally data is either administrative, i.e. government-generated or from independent surveys. It is always prudent to use administrative data, and if it is not reliable, or not available, then it is incumbent on the UN system invest resources to strengthen the national data information system. With regards to survey data, a lot of care needs to be taken in selecting any sources that are periodical. For example, some surveys are conducted every 5 years or even 10 years – it therefore would not make sense to use these as sources if the programme cycle falls outside these timelines. 


Tag: Effectiveness Monitoring and Evaluation Policies & Procedures Project and Programme management Results-Based Management Theory of Change UN Country Team Data and Statistics

5.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

4.2.1. Result Group 1. Agriculture

The UN Result Group for Agriculture focused on the UNDAF Outcome 1: By 2020 Ethiopia will achieve increasingly robust and inclusive growth in agricultural production and productivity and increased commercialization of the agricultural sector. Based on review of the joint biennial work plan (2016 – 2018), six UN agencies were contributing to this outcome, with FAO and UNDP as the co-conveners. However, analysis of the UNDAF annual reports shows that two of the participating UN agencies (ITC and UNIDO) did not undertake any interventions under this outcome. UNDAF consolidated reports show significant results, including; (a) distribution of 400 MT of improved seed to 169,000 households, of which 33,490 (20%) are women headed households; (b) establishment of Input Voucher System (IVS), which benefitted a total of 3.5 million smallholder farmers who accessed 4.7 million quintals of agricultural inputs through the system; (c) establishment of the Interactive Voice Response and SMS platform, which benefited over a million smallholder registered callers to obtain real-time agronomic information. In 2017/18, the UN continued support to Small Holder Farmers (SHFs), Pastoralists and Agropastoralists (PAP) reaching a total of 448,855 beneficiaries out of who 218,540 SHFs adopted new crop seed varieties and applied improved cropping practices such as Climate Smart technologies, improved agricultural technologies and fishery resources management; while 230,315 PAPs benefitted through livestock feeds and animal health interventions, including Community-Managed Disaster Risk Reduction (CMDRR), developing community investment plans, conducting Agro-Pastoral/Pastoral Field School (APFS) training. The UN also supported the GoE to develop and implement adopting various agricultural strategies, regulatory frameworks and standards, including notably: - Camel Milk Value Chain Development Strategy for pastoral areas, - Resilience strategy and pastoral resilience approach, and - National post-harvest handling strategy. 

The above results do not sufficiently reflect the disaggregated contribution of the UN, and likely reflect the national results achieved under the GTP. However, within their respective programmes, UN agencies contributed to these results, as illustrated by the following examples. The UN supported integration of Climate-Smart Agriculture (CSA) into the current Program Investment Framework (PIF) to MoANR and Ministry of Livestock and Fisheries (MoLF), including through three studies on the integration of CSA in the tertiary education system of Ethiopia, cost-benefit CSA technologies and practices and private sector engagement in promoting CSA. The UN also supported a programme on flood mapping and development of strategies and plans to meet the energy needs of refugees and host communities in Gambela Region, which was reported to have reached 15,000 households. The UN supported Nutrition Sensitive Agriculture has been promoted through establishment of 75 demonstration gardens at 50 Farmer Training Schools (FTCs) and 25 schools. Capacity building was also provided for 47 Development Agents in the area of household food reserve management, good nutrition practices, including dietary diversity and complementary food preparation. In addition, preliminary analysis on agro-infrastructure in the Agro-Commercial Processing Zone (ACPZ) of the Central-Eastern Oromia Integrated Agro-Industrial Park (IAIP) were also undertaken.  

The above are examples of the actual contributions of the UN system, which need to be captured through the UNDAF results, monitoring and evaluation framework. Overall, 14 out of 22 output indicators (64%) were on track. At outcome level, only one indicator has been reported as achieved; there was no data reported for the other 5 indicators in 2016 and 2017. Some key issues for consideration by the RG are noted below. • Outcome indicators. All the six outcome indicators (1.1 to 1.6) were not reported in both years against indicators. The reports for both years have narrative information instead of quantitative data as per the indicator targets. The RG should establish whether these need to be maintained, modified or completely removed depending on data availability. The RG may also need to reconsider indicator 1.5 because there do not seem to be any outputs supporting fisheries specifically. • Misreporting: Mismatch between measuring scale, where the indicator measure is in # and reporting is in % (e.g. 1.1.2). Indicator 1.2.1 has irrelevant information reported: instead of “% increase of value added commodities” as per indicator, the reports narrate about capacity building and investment. • Blank data. The status of indicators should be reported even if no activities were undertaken; there should be no need to leave blank as is the case for indicators 1.2.4 and 1.3.4. • Duplication. The same information was provided for indicators 1.1.4 and 1.3.3 in the first year reporting despite that the indicators are different. • Vague indicators. Indicator 1.4.1 and 1.4.4 are vague and difficult to measure, For example, 1.4.1 “number of rural women accessing integrated agricultural services” is vague and not specific. 


Tag: Agriculture Effectiveness Women's Empowerment Monitoring and Evaluation Results-Based Management Country Government Private Sector UN Agencies Inclusive economic growth Trade and Development Data and Statistics

6.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

4.2.3. Result Group 3. Disaster Risk Management

Result Group 3 coordinated the results for the UNDAF outcome 3: By 2020, the Ethiopian people, particularly in disaster prone areas are resilient, have diversified sources of income and are better able to prepare, respond to and recover from emergencies and disasters. The UNDAF itself shows nine UN agencies participating, with WFP and UNDP as co-conveners. However, a review of the UNDAF biennial work plan (2016-2018) shows only three UN agencies (IOM, UNDP and UNHCR) with committed funding. As noted previously, much of the UN work undertaken under this RG was aligned to the NWoW. Some notable and illustrative examples are shown below. The UN undertook assessment of displacement sites in 481 kebeles, 173 woreda and 44 zones to identify the multi-sectoral needs of the communities using the Displacement Tracking Matrix (DTM). In Somali region, a durable solution Working Group for IDPs was established, comprising of regional bureaus, UN agencies and international NGOs and its terms of reference include monthly coordination meetings.

Three Disaster Risk Management (DRM) units (i.e. NDRMC at Federal level and Somali and Gambella DPPB/O) track the Emergency Supplies/Non-Food Items (ES/NFI) stock and needs and shares the data at federal and regional level in Gambella and Somali during regional ES/NFI cluster meetings that are held monthly at the region and twice at federal level. Through this support, the UN distributed 51,601 ES/NFI kits to targeted beneficiaries, comprising 17% men, 18% women, 33% boys and 32% girls. The UN also supported youth groups from 210 households (130 refugees and 80 host communities) representing 1,155 individuals in Melkadida, Bokolmanyo and Dollo Ado camps and nearby hosting kebeles, and provided them vocational and life skills training. Livelihood support through income generating activities (IGAs) was provided to a total of 1,210 households (800 refugee households and 410 host community) representing 6,050 individuals in Gambella region; 530 households (400 refugee households and 130 host community) representing 2,650 individuals in Benshangul Gumuz region . 

In 2016 alone, through the joint food pipeline of NDRMC and WFP, the UN’s relief assistance reached 7.8 million beneficiaries in nine months of distributions. WFP alone distributed 218,671 MT of food and US$ 26,720,913 cash transfer. A further 263,603 MT of food and $3,340,218 cash transfers were provided in 2017. Despite these achievements, some issues that the RG should address to strengthen its reporting are outlined below.

• No progress report received for first year: The evaluators were unable to get the progress report for the first year. • Broad indicator: Indicators should be specific and achievable. However, it will make measurement difficult if we specify broad indicators, such as 3.3 which seems to be a global indicator. • Inconsistent indicators. Some indicators with measurement units in % were reported as #; e.g. 3.1.2 and 3.3.3, while in other cases such as 3.3.2, the measurement unit is % but the target is in #. • No progress reported: Progress was not reported for the following indicators: outcome indicators 3.1 – 3.3, and output indicators 3.2.1, 3.2.3, 3.4.2, 3.4.3. • Similar indicators. Some of the indicators seem to measure the same thing, such as 3.3.2 and 3.3.3 and these could be merged because what was reported was closely similar. 3.2.4 and 3.2.5 could also be merged. 


Tag: Disaster risk management Vulnerable Effectiveness Results-Based Management Theory of Change UN Country Team Displaced People Resilience Capacity Building Poverty Reduction Social Protection Coordination

7.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

4.2.5. Result Group 5. Climate Change Results

Group 5 brings together seven UN agencies to coordinate their work towards UNDAF Outcome 5: By 2020 key government institutions at federal and regional level are better able to plan, implement and monitor priority climate change mitigation and adaptation actions and sustainable natural resource management. The RG members engaged in FGDs said that coordination was a challenge, which was beyond the country level, as it was impacted by the different corporate requirements that UN agencies had to adhere to from their respective headquarters. They also noted that some UN agencies had initially pledged to contribute to the outcome but had not done so when implementation started. Despite these challenges, they observed that there were a number of ongoing joint programmes, such as the United Nations Programme on Reducing Emissions from Deforestation and Forest Degradation (UNREDD), although most were global and did not start during this UNDAF cycle. Out of 28 output indicators, thirteen (46%) were reported on track, 12 were off track (42%), while three were in progress. The outcome has 3 outcome indicators of which one was reported on track and two were offtrack. However, the RG also reported notable achievements, including, (i) support for the development and publication of the national Forestry Action Plan, (ii) development of the National Waste Management Strategy and National Short-lived Climate Pollutants Unit. The UN was also supporting the waste management and urban greenery initiatives, thereby contributing to the overall objective to reduce carbon emission. 

Some of the reporting challenges observed with regards to the indicator framework include: • Omission of indicators. Eight indicators were not reported for both years (5.1.5, 5.2.1, 5.2.3, 5.2.4, 5.3.4, 5.3.5, 5.3.6 and 5.3.7) and it is not clear whether this is due to lack of data or that the indicators had become irrelevant. • Revision of indicators. While the RG has the prerogative to review and revise indicators, there has to be some consistency. Indicators cannot be changed back and forth from one year to the other as was the case for indicator 5.3.5 and indicators 5.4.1 – 5.4.4. New indicators for output (5.3.8 and 5.3.9) were also added in the second year reporting. • Narrative reporting. There was a lot of narrative reporting against quantitative indicators.


Tag: Climate Change Adaptation Climate change governance Natural Resouce management Health Sector Nutrition Joint UN Programme Policies & Procedures Results-Based Management Refugees Coordination Data and Statistics

8.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

4.2.7. Result Group 7. Water, Sanitation and Hygiene

Five UN agencies coordinated their work under Result Group 7, with UNICEF and UNOPS as co-conveners. While progress for the outcome indicator was difficult to ascertain due to lack of disaggregation of both baseline data as well as the planned targets, some of the UN’s notable results are outlined below.  

By end of 2016, 38.4 percent of primary schools had access to clean water supply, and 86 percent of primary schools were reported to have some kind of latrines. However, only 3.2 percent of schools had access to full package of WASH facilities. According to a 2017 WASH KAP survey, 23 percent of women and 13 percent of men reported that they wash their hands with soap or ash at all the three critical moments (before eating, after defecation / using a toilet and before handling and/or preparing food) during the preceding 24 hours of the survey. The UN also targeted specifically the refugee population, and in that target group, 75 percent of the camps were above 20 liters per person per day whilst household latrine coverage was on average 57 percent.  

Although functional WASH coordination mechanisms were reported to have been established at Federal and regional levels, overall only 40 percent of output indicators (6 out of 15) had made progress greater than 50 percent of planned targets. Five indicators (33%) were reported off-track with progress below planned targets. Some specific issues requiring the attention of the RGs are listed below

• Measurement unit. There were inconsistences in the reporting of some indicators, such as 9.4.1, 9.4.2, and 9.4.3 in which the first year reports progress in % terms and in # terms in the second year. Other indicators, such as 9.3.3, 9.3.4, 9.4.1, 9.4.2, 9.4.3 also have inconsistence of measurement unit. • Target modification. The target for indicator 9.1, is 83%, but it was changed to 61% in the reports. The reports also specifically refer to drinking water only, while the indicator itself is for WASH, which includes sanitation. • Adding outcome indicators. The UNDAF outcome has one indicator (9.1), but an additional indicator surfaced in the 2017/18 report even though it is was not there in AWPs. • Progress duplication. The same narrative was reported as progress for both 2016/17 and 2017/18 for output indicators 9.1.4, 9.2.1, 9.3.1, 9.3.2, 9.3.3, 9.3.4. • Vague indicators. The outcome indicator (9.1) measures ‘% of populations using safe and adequate WASH services disaggregated by rural and urban areas’. However, the baseline is not adequately disaggregated. 


Tag: Vulnerable Drinking water supply Sanitation Effectiveness Health Sector Results-Based Management UN Agencies Education

9.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

4.2.9. Result Group 9. HIV

The joint biennial work plan lists six UN agencies as coordinating their work under Results Group 9 on HIV. Some key informants observed that this did not reflect the actual reality, because the UN Joint Team on HIV is globally composed of 11 UN agencies (including in addition to the six: ILO, UNDP, UNESCO, UNODC, UN Women and World Bank), with each one having a specific mandate and role. The informants also observed that the work plan had 30 indicators, whereas the global joint programme had only 11 indicators. There was generally lack of data, which made it difficult to measure progress based on all 30 indicators, while also the UNDAF had adopted national targets which could not be achieved by the UN. Overall however, UN Agencies reported positive progress in various areas. Through UN advocacy and support, the minimum HIV prevention service package was adopted in educational institutions, while also procurement and distribution of HIV protective kits continued. The proportion of at-risk groups, such as people living with HIV and AIDS (PLWHA) accessing Anti-retroviral treatment increased, as did the number of people living with HIV (PLHIV) who received livelihood and income-generating activities (IGA). While other planned activities were reported to have been delayed, some specific issues related to reporting against the indicator framework are outlined below. 

• Missing indicators. There was no data reported against 3 outcome indicators (8.1, 8.2 and 8.3) as well as seven output indicators (8.1.4, 8.1.6, 8.1.7, 8.2.1, 8.2.2, 8.4.2, 8.4.4). The practice of leaving blank spaces is not helpful. • Inconsistent measures. A number of indicators have inconsistent measuring units, such as for example 8.2.3 which provides an indicator to be measured in both % and numbers, while the reporting eventually ended up in % only. There are also cases where the indicator measures a number, but baseline, target and reporting is in %, (8.2.4). • Overlapping indicators: Indicator 8.3.3 provides % of children <15 while 8.3.4 provides % of adolescents 10-19 years, which may potentially result in double counting. Indicator 8.3.2 also broadly refers to % of adults, without specificity about their age range. 


Tag: Conflict resolution Social cohesion Data and Statistics Gender Equality Gender Mainstreaming Women's Empowerment Civic Engagement Election Human rights Justice system Rule of law HIV / AIDS Policies & Procedures Results-Based Management UN Agencies

10.

4.2. Effectiveness of UNDAF Implementation and Results Achievement (Continuation)

4.2.11. Result Group 11. Data and Demographic Dividends 

This results group coordinates UN work towards UNDAF outcome 13: By 2020, national and sub-national institutions apply evidence-based, result-oriented and equity-focused decision making, policy formulation, programme design, monitoring, evaluation and reporting. While the UNDAF shows that six UN agencies initially pledged to contribute to the outcome, but two of them (UNCDF and UNCTAD) did not participate. In the course of implementation however, one UN agency (IOM) has begun to participate under the RG while another agency (UNHCR) has expressed interest and relevance to join the RG in the future. 

The outcome has one indicator, which does not have a baseline. While performance can be measured for as long as there are performance targets, the absence of a baseline makes it difficult to track actual development changes over time. In addition, most of the qualitative indicators used at output level are difficult to measure, either due to lack of data or vagueness of the indicators. Nonetheless, the RG reported some notable progress, including support for establishment of functional civil registration and vital statistics system in most woredas in the country; as well as ongoing comprehensive and disaggregated national data and household surveys and census through electronic-based platforms. 

Some discrepancies on the indicator framework are noted below: • Inconsistent measures. The outcome indicator is difficult to measure since the measurement unit is in numbers but the indicator itself is qualitative. Also indicators 13.1.1 and 13.2.2 have inconsistent measurement units. • Missing baseline data. There is no baseline data for indicators 13.1 and 13.3.3. 


Tag: Effectiveness Gender Equality Gender Mainstreaming Gender-Based Violence Women's Empowerment Capacity Building Inequalities Youth Data and Statistics

11.

4.3. Efficiency of UNDAF Implementation and Coordination

In order to respond to the TOR, efficiency has been assessed from a perspective of the UNDAF’s capability as a platform for DaO as well as for obtaining and use of resources. The evaluators were also cognizant that the UNDAF was formulated fully in line with DaO principles as noted therein: ‘in order to operationalise the DaO approach, the UNCT has developed a strong DaO governance framework in line with the UNDG Standard Operating Procedures for DaO…”(p 15)

Finding 9: The UN has made progress towards harmonising operations

In line with the UNDAF in Ethiopia for 2016-2020, the Operations Management Team (OMT) and BOS Committee developed the BOS 2.0 2016-2020 with the objective of applying a comprehensive approach to the UN’s business operations in Ethiopia with the aim of continuing to enhance quality, effectiveness and efficiency of joint common services across UN System Agencies in Ethiopia.

Through implementation of the BOS 2.0, the OMT contributed to increased coherence and harmonisation of UN operations, including notably the following:

- UN Organizations were able to save US$ 2,460,254 from using common Long Term Agreement (LTAs), conducting joint training such as the Certificate in Public Procurement (CIPS), PRINCE 2 Project Management and Competency-based Interview Skills (CBIS), as well as advocacy for an arrangement to re-claim the value-added tax (VAT). - As a result of joint training in CIBS by 36 programme and operations staff, savings of US$ 125,532 were realised. - LTAs for vehicle insurance and printing services were finalized. - Recruitment time for staff and consultants was reduced through the CBIS training which was attended by 71 UN staff from 21 UN agencies. - In order to strengthen Delivering as One (DaO), the UNCT endorsed the introduction of the Model Award Framework in June 2017 to acknowledge individuals and teams on the basis of their efforts, performance and accomplishments in inter-agency work. - In September 2017, the report on the Macro Assessment valid until 2021 was approved by the Ministry of Finance and Economic Cooperation (MoFEC)

Based on key informants, one of the factors that contributed to successful implementation of the BOS 2.0 culminating with enhanced coherence of UN operations was the provision of dedicated secretariat to the OMT by the UNRCO. The OMT Secretariat is a part of the RCO under the Operations pillar of the Delivering as One (DaO) structure. The Secretariat has three staff members: 1 international staff (P 2) and 2 national staff (SB 4). The evaluators noted that the UNCT was fully committed to funding the BOS 2.0 (Table 7). The OMT work plan outlined its funding requirements for the four-year UNDAF period as $1,780,930, and based on the funding received in 2016-2017, the OMT’s budgetary requirements are on track. Table 7: Resource requirement for the BOS 2.0

The evaluators also noted however that out of the 28 members of the UNCT, 23 were currently participating in the BOS 2.0. According to the OMT, the UN agencies that do not participate in any / some of the BOS arrangements had valid reasons or limitations:

- IFAD - their operations (HR, Procurement, M&E etc.) are implemented by the Government of Ethiopia, - UNODC - there was no country representative appointed, while also their local staff did not yet have formal contracts, - World Bank and IMF - they are not part of UNDAF and do not participate in the common UN system, - ITC – It is basically a non-resident agency (NRA), with only two local staff in-country, - UNISDR - Also a NRA with one international and one national staff.


Tag: Coherence Efficiency Resource mobilization Harmonization Human and Financial resources Joint UN Programme Operational Efficiency Project and Programme management Results-Based Management Capacity Building Coordination Operational Services

12.

4.3. Efficiency of UNDAF Implementation and Coordination (continuation)

Finding 11: The UN has adequate structures for coordinating UNDAF implementation and DaO

The context of DaO in Ethiopia started when the country decided to adopt it as a self-starter. This is important to acknowledge because it means different things to different people. For some, it means that as a selfstarter, it is not bound to fully comply with all the principles outlined in the SOPs for the DaO Pilot Countries. For others however, it means that since Ethiopia had an option not to ‘deliver as one’ but chose to, then it has to demonstrate its full commitment and compliance with the SOPs. This discussion is beyond the scope of this evaluation. And so is the discussion about the complicated nature of UN DaO arising from the way in which Agencies, Funds and Programmes (AFPs) operate with their headquarters and with their donors. What is important to note is that Ethiopia is a DaO self-starter and therefore aspires to enjoy the benefits thereof, including increased effectiveness from enhanced coherence and harmonisation of procedures, as well as from reduced duplication and transaction costs.  

The UN coordination structure has all the relevant and essential structures required for UNDAF coordination and DaO (Figure 2). This is not to say that the structures work effectively, but to acknowledge that the structures are appropriate for UNDAF coordination.Figure 2: UNDAF and DaO Coordination Structure. Joint UN-GoE Steering Committee. The High-level Steering Committee (HLSC) was established ‘to provide strategic direction and oversight of the UNDAF process throughout its implementation cycle’. It is co-chaired by the State Minister of MoFEC and the UN Resident Coordinator (UNRC), with its membership comprising representatives of GoE, UNCT and development partners rotating annually. The HLSC last met in June 2017 to approve the joint biennium work plan and has been dormant since. According to key informants, some of the reasons why it has been dormant include: • Its membership also attend other coordination, such as the Development Assistance Group (DAG) and Sector Working Groups, which overwhelmed them, • The idea of rotational membership was not conducive for continuity, • There were too many changes of personnel both in GoE and in the UNCT during the 2-year period.

UN Country Team. The UNCT comprises of the Heads of Agencies (HoA) of all UN entities accredited to Ethiopia under the leadership of the UNRC. It oversees the development and implementation of the UNDAF and the work of all inter-agency working groups and results groups. The UNCT meets weekly. The evaluation noted however that the UNCT did not prioritise the UNDAF in its routine, and did not adequately hold thenter-agency working groups and results groups to account. For example, the practice in earlier days for the PMT and OMT to report back to the UNCT in alternate sessions was discontinued in 2017.

Programme Management Team (PMT). The PMT is the senior programming coordination structure, which ensures a commonality of approach across the UN system as well as adherence to DaO principles. The chairperson of the PMT is required to report regularly to the UNCT on progress towards UNDAF results. According to information obtained during FGD, the PMT members were overwhelmed with too many meetings, and they did not see the UNDAF’s added-value, mainly because of its structure, which they felt was a ‘listing of what UN agencies do, and not what the UN should do together’. In addition, they also noted that the UNCT did not require accountability, and this contributed to general perception that UNDAF was not that important. The PMT exercises oversight of the Results groups and the M&E Working Group, but they only call on them when it is time to produce annual work plans and joint annual reports. Besides this, they have no other formal interaction with the Results Group or M&E Group specifically for the UNDAF. 


Tag: Efficiency Gender Equality HIV / AIDS Harmonization Joint UN Programme Oversight Policies & Procedures Project and Programme management UN Country Team Youth Coordination Operational Services

13.

4.3. Efficiency of UNDAF Implementation and Coordination (continuation)

Finding 12: The UNRCO has adequate capacity and structure to support UNDAF coordination and DaO

The RCO is the institutional mechanism for effective functioning of the UNRC, and also provides coordination support to UNDAF structures, including support to the UNCT and its working groups. The RCO in Ethiopia is quite big relative to other countries (Figure 3), with a staff complement of 14 staff, of which six are international and eight national staff. Two positions of Development Effectiveness Specialist and SDGs Support Officer were vacant at the time of the review; as was also the UNRC. Figure 3: RCO Structure

In its coordination support role, the RCO provides dedicated secretariat support to the PMT and Results Groups, the OMT and UNCG. In the evaluators’ opinion, the RCO is quite well positioned and structured to provide adequate support for UNDAF coordination and DaO. In addition to coordination support, the RCO also provides analytical and policy support through the following staff functions. 

• The Head of Office provides leadership of the RCO team, and is the DaO focal point for UN System in Ethiopia, including engagement with Government and support to the HLSC. Also responsible for advising the UNRC/UNCT on the strategic agenda, facilitating joint resource mobilization initiatives, and chairing the JPAT, supports HLSC, UNCT and coordination with DAG. • Peace and Development Advisor (PDA) - provides support to the UNRC/UNCT on conflict analysis in the context of the UN’s peace building agenda.

• Policy and Integration Advisor – provides support on joint policy analysis, including nexus issues.

• SDGs Support Officer – support to SDGs sensitisation and coordination, as well as implementation of the UN external outreach strategy, and focal point for specific outreach initiatives and Youth Task Force activities.

• Development Effectiveness Specialist - support to inter-agency programme collaboration, including joint programme (JP) initiatives and management of JPs. Also provides support to inter-agency capacity building efforts for advancing DaO initiative, as well as support to missions/visits of the RC and partnership building. 


Tag: Efficiency Human and Financial resources Joint UN Programme Project and Programme management UN Country Team Coordination

14.

4.4. Sustainability of UNDAF Results and Processes

This section addresses the evaluation criteria for sustainability, and assess the probability that UN processes and results would be likely to continue after the end of programme funding. In the context of Ethiopia, the question of sustainability has even more importance given the country’s vision to transition towards middle income status. This means that the country should establish the necessary systems and infrastructure for basic social service delivery, and hence systems development and capacity building take on increased importance in such a context.

Finding 13: UNDAF implementation through government structures both at federal and regional levels promotes national ownership and capacity building

The UN uses the national implementing modality (NIM) at federal and regional levels. In practice, this means that when UN projects are developed, relevant government line Ministries and departments are engaged as the implementing partners (IPs), and funds are either transferred directly to the coordinating line Ministry or through the regional Bureaus of Finance and Economic Development (BoFED) to the sector Bureaus. An analysis of the UNDAF outputs shows that a majority of them (almost all) seek to strengthen national technical capacity or systems. The output indicators however do not seem particularly designed to measure the UN’s contribution towards this capacity building. To the extent that UNDAF indicators and targets are direct GTP targets, they reflect more of direct service delivery results. In the UNDAF the UN noted that “…making the indicators less ambitious, especially at the outcome level, in order to allow for better attribution of outcomes to UN programme work” (p 16). 

Based on key informant interviews, it was noted that the UN lacks a shared understanding of the ‘contribution’ versus ‘attribution’ theory of results-based management. Most discussants have the generalised notion that the ‘UN is accountable for outputs, but only contributes to outcomes’. While this is not necessarily incorrect, it should however be taken in its appropriate context. In the evaluators’ opinion, as individual entities, indeed UN agencies only contribute to UNDAF outcomes, but as a collective, the UNCT should be accountable for the delivery of planned results at the outcome level. This requires the UN to develop SMART (Specific, Measurable, Achievable, Realistic and Time-bound) indicators. The UNCT may wish to elaborate and advise a specific position and instruction for Results Groups in order to harmonise this. As noted in Chapter 2 above, there was no data collection undertaken at regional level. However, some of the regional stakeholders participated at the mid-term review presentation of preliminary findings that was undertaken at the end of the primary data collection. They observed that coordination between the government and the UN at regional level was weak. Given the country’s geographic spread, as well as the diversity of issues between the regions, the UN may consider setting up regional coordination offices. This will contribute to strengthen UNDAF coordination since regional governments are one of the major implementation partners.


Tag: Disaster Risk Reduction Sustainability Local Governance Health Crises Implementation Modality Joint UN Programme Ownership Results-Based Management Strategic Positioning Theory of Change Country Government UN Country Team Humanitarian development nexus Capacity Building Social Protection Coordination Technical Support Leaving no one behind

Recommendations
1

6.2. Recommendations

The following nine recommendations include both strategic and operational considerations that the UNCT should decide on for the remaining period of UNDAF implementation as well as for the next UNDAF. These recommendations are not intended as an alternative, but rather as additions to what the Results Groups have recommended through their annual reporting.

Recommendation 1. The UNCT should give specific directions and instructions to ensure that the next UNDAF is considerably inclusive and simplified. This may include:The next UNDAF should be informed by a UN-led comprehensive common country assessment (CCA), b) The next UNDAF should contain no more than 5 collective outcomes that enhance the UN’s development, humanitarian, human rights and peace building agenda.

2

Recommendation 2. The HLSC should consider and make the following decisions to enhance UNDAF implementation and UN results in the remaining period of UNDAF implementation:

a) Acknowledge that the current UNDAF structure and content has not been conducive for Results Groups to undertake joint programming, implementation and reporting against indicators

b) Results Groups in collaboration with the PMT and M&E Group should make necessary changes through the joint AWP, including:

(i) Revision, removal and addition of UNDAF outputs,

(ii) Revision, removal and addition of UNDAF outcome and output indicators.

3

Recommendation 3. The UNCT should give specific directions and instructions to ensure that the next UNDAF is considerably inclusive and simplified. This may include:

a) The next UNDAF should be informed by a UN-led comprehensive common country assessment (CCA),

b) The next UNDAF should have outcome-level results only, with outputs articulated in the joint AWPs,

c) The next UNDAF should contain no more than 5 collective outcomes that enhance the UN’s development, humanitarian, human rights and peace building agenda.

4

Recommendation 4. The UNCT should commit to strengthen UNDAF implementation and its use as the principal planning document for their respective country programme. This commitment should include at a minimum:

a) Ensuring that UNDAF implementation is included in individual performance appraisal for key staff (e.g. PMT staff, senior programme staff),

b) Ensuring that every UN agency contributes in at least one joint programme or joint initiative.

5

Recommendation 5. The UNRC/UNCT should take necessary measures to establish the One Fund by enhancing joint resource mobilisation. As a starting point, the UN may leverage on specific joint initiatives or flagship programmes such as for example:

a) Joint UN low-land strategy,

b) Joint UN initiative on resilience and reducing vulnerability and inequality.

6

Recommendation 6. The UNRCO should simplify the UNDAF reporting template (Figure 4);

a) to make it more user-friendly, and

b) to adequately reflect and report performance linked to the UN’s core principles and standards, including reporting on SDGs. The structure of the template may be in two parts, including a narrative section and matrix.

See in the report- Figure 4: Proposed UNDAF Reporting Templatetable 

7

Recommendation 7. The UNRCO should develop an online UNDAF training module for all UN staff. The module should be developed along the lines of the mandatory Basic Security in the Field Manual, with prescribed validity of up to 3 years. The UNCT should commit to ensuring that the training is mandatory.

8

Recommendation 8. The UNCT should ensure that programme staff are not too overwhelmed with coordination work by limiting and merging some of the coordination platforms with Results Groups that have similar focus, such as for example the Youth Task Force. 

9

Recommendation 9. The UNCT should consider establishing Regional Coordination Offices. The UN may consider piloting these in regions where there is already larger presence of UN agencies, such as Somali and Amhara regions.

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org