Evaluation Detail

Final evaluation Ministry of Economy and Planning Project

Evaluation Plan:: 2017-2022, Saudi Arabia
Evaluation Type:: Final Project
Planned End Date:: 01/2022
Completion Date:: 01/2022
Status:: Completed
Management Response:: No
Evaluation Budget(US $):: 35,000

Document	Type	Language	Size	Status	Downloads
210617 Evaluation TOR MoEP-updated final.docx	tor	English	60.08 KB	Posted	209
220106 Final Evaluaiton report - MWT.docx	report	English	962.65 KB	Posted	155

Title

Final evaluation Ministry of Economy and Planning Project

Atlas Project Number:

00116709

Evaluation Plan:

2017-2022, Saudi Arabia

Evaluation Type:

Final Project

Status:

Completed

Completion Date:

01/2022

Planned End Date:

01/2022

Management Response:

Yes

Focus Area:

1. Poverty
2. Others

Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)

1. Output 2.1.2 Capacities developed for progressive expansion of inclusive social protection systems

SDG Goal

Goal 10. Reduce inequality within and among countries
Goal 16. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels
Goal 5. Achieve gender equality and empower all women and girls

SDG Target

10.1 By 2030, progressively achieve and sustain income growth of the bottom 40 per cent of the population at a rate higher than the national average
16.6 Develop effective, accountable and transparent institutions at all levels
5.5 Ensure women’s full and effective participation and equal opportunities for leadership at all levels of decision-making in political, economic and public life

Evaluation Budget(US $):

35,000

Source of Funding:

Project budget

Evaluation Expenditure(US $):

27,817

Joint Programme:

Joint Evaluation:

Evaluation Team members:

Name	Title	Email	Nationality
Cosette Maiky	Independent Evaluator	cosette.maiky@gmail.com

GEF Evaluation:

Key Stakeholders:

Countries:

SAUDI ARABIA

Lessons
1.	In the extremely specific realities of the KSA, the results-oriented approach and continuously monitoring of internal and external supporting and constraining factors affecting operations are key factors to ensure the conception and implementation of relevant interventions.
2.	The importance of strategic partnerships between international organizations and national partners on advancing knowledge and integrating evidence at the policy-making level is critical and essential.
Findings
1.	Relevance *The relevance of the project is rated as ‘Acceptable’***. Problem analysis, prioritisation, and adaptability to emerging needs have been performed throughout the implementation period. Most respondents affirmed that the implemented interventions were aligned to the mandate, strategy, function, roles, and responsibilities of key actors (UNDP and MEP respectively), and well-aligned with national and global priorities (Vision 2030, SDGs). In addition, it was found that the design of the implementation strategy of the project was coherent and consistent with the Country Programme Document during the Programme Cycle 2017-2021.
2.	Effectiveness *The effectiveness of the project is rated as ‘Acceptable’**. Ideally, it would have been extremely useful to interview the individuals (National Project Managers) who managed the project’s interventions from the UNDP and the MEP. This step could have helped in gaining more comprehensive insights about the outcomes. However, this step was not possible since these people were not on board anymore. This is considered to have a significant impact on the consistency of the assessment of progress under the effectiveness* criterion, which exclusively relied on the insights gathered from both secondary and primary research. Some of the reported achievements (mostly at the activity level) in the 2020 Annual Report: Experts assigned to support the Minister’s office, particularly in areas of report writing, review of studies and technical reports, presentation of analysis of issues and results, as well as provision of opinions, notes, and advice. Contribution to economic country reports, including partnership opportunities proposal with the KSA, for official high-level State visits or missions abroad. Contribution to the KSA participation in the WEF (Feb 2019), the Islamic Summit, Arab Summit, CGC Summit, etc. Advising the Saudi delegations in diverse events, i.e. participation in the ESCWA high-level meeting in Amman, etc. Participation in the Citizen’s Account Committee activities. With a marked flexibility and ability to adapt to unexpected threats or risks – especially since the onset of the COVID pandemic - the team managed to achieve the desired results during the implementation period. However, the quality of the existing documentation did not provide enough evidence to verifiably demonstrate the effectiveness of planned outputs and their plausible contribution to the expected outcome. The adopted actions and strategies were mostly relevant to achieving the expected results in the concerned areas during the implementation period. The elements detailed below in terms of professionalization, strengthening quality and accurateness, engaging experts, were identified: UNDP provided advice and technical assistance to bring about policy changes towards the desired goals, UNDP called upon Arab and non-Arab experts for contribution to the produced knowledge materials (although not as systematically as it could have been), During the last quarter of the 2-year implementation period, UNDP managed to reinforce and diversify partnership strategies (called upon for synergies between the UNDP, the MEP and the General Authority for Statistics), Customized capacity development activities in a context-specific and culturally sensitive way.
3.	Efficiency *The efficiency of the project is rated as ‘Needs mild improvement’***. The project’s annual report followed an activity-based reporting style that did not sufficiently emphasize the multidimensional results of the project. According to interviewed stakeholders, most interventions (except recruitment processes) were well and timely executed with no significant reported delays. According to most interviewees, there is a need to standardize and systematize content management processes and mechanisms, for implementing partners to significantly reap synergy benefits from well-defined reporting, documentation and monitoring and evaluation processes and mechanisms. Efficiency poses the question whether, given the budget, the specified outputs could have been achieved at a lower cost. The Evaluator did not have any comparative data to make a fair cost-efficiency judgment. Against the noted challenges, all respondents acknowledged that the project had a “large margin of flexibility and adaptability” both during planning and implementation. Starting the 4th quarter of the implementation period, UNDP team started building on lessons learned during the cycle and started shifting towards more strategic and system-level thinking. The dedicated staff have shown to be resilient to the rapid and drastic changes by managing to speed up the coordination pace prior to starting the implementation of the new Agreement. While this observation can be of great significance by means of change, it remains limited to the particular context of the Evaluation exercise and the limited generalizability (due to the qualitative nature of a considerable part of results), and hence, needs further investigation. It can be expected that as the project continues through the newly signed off Agreement, the ability of the team to expand through the adequate recruitment of a Chief Technical Advisor – who can bring to the table the comparative and competitive advantage of UNDP as compared to private management consulting firms -, and to adopt an implementation approach that is based on a solid system-level analysis will most likely translate into strategic results and policy-focused interventions. As work mostly included elements of soft assistance (e.g., standards development, training, policy advocacy) it makes the application of conventional efficiency indicators to these areas not feasible (i.e. procurement of assets, comparative analysis to alternatives…). Communication tools were insufficiently utilized during the implementation period of the project (no quarterly reports or briefs, presentations, visibility products were made available to the evaluator). As such, causality and attribution are yet to be validated and consolidated. The project did not have a robust Monitoring and Evaluation (M&E) architecture in place, which has evidently hindered the systematic documentation of progress and achievements. Although the project lacked systematic mainstreaming of cross-cutting issues, most respondents acknowledged that along the implementation period, a lot of effort was exerted to reflect on specific issues that may be experienced by women who newly entered the labor market, and the societal forces that shape power relationships. However, considering the documentation of the project (project documents, progress reports), one would find that it does not reach the level of systematic incorporation.
4.	Sustainability *The sustainability of the project is rated as ‘Acceptable’***. The available data could not provide a consistent characterisation of how UNDP’s team utilized relevant strategies in pursuing its catalytic role with measurable success indicators and an operational “exit” to ensure that the MEP can sustain and expand their activities independently - or with minimum support. The provision of capacity building, capacity development and technical assistance to the MEP introduced an element of sustainability into the interventions implemented. Moreover, the project in many interventions depended on trainers who progressively became agents of change adding to sustainability (a senior expert on board has been actively engaged since 2006). Despite the progress in the achievement of a number of activities towards intended results, the contribution of UNDP cannot be quantitatively measured. The project provided a number of achievements which benefited a wide number of beneficiaries. However, the causal relationship attributing this progress to the interventions implemented is challenging and almost impossible to be proven. The trade-off between the maturity and extent of impact, time, and cost of the interventions remains a challenge to the project. The previous phases of the project were able to provide an ample number of demonstrations and pilots that could guide future successful interventions. Consequently, the approach through which some interventions are implemented on the ground calls for a change in the model’s mandates in future phases.
5.	Conclusion The Evaluation conclusion was defined based on the answers to evaluation questions and associated findings: the project is an Acceptable project that has created outputs and results with impact potential for successor projects, should they be premised on lessons learned and good practices, along with all other pillars since the early stages of project conception. Problem analysis, prioritisation, and adaptability to emerging needs have been performed throughout the implementation period. The implemented interventions were aligned with the mandate, strategy, function, roles, and responsibilities of key actors (UNDP and MEP respectively), relevant to the originally identified technical priorities and needs, and well-aligned with national and international priorities (Vision 2030, SDGs). The design of the implementation strategy of the project was coherent and consistent with the Country Programme Document during the Programme Cycle 2017-2021. In its second year, the project witnessed a notable performance in terms of project management and resource efficiency. The distribution of tasks in the CO increased the efficiency of management and contributed to the swift responsiveness of the project to contextual shifts. There is room for improvement in the project’s logical framework and progress indicators. The project did not have a robust Monitoring and Evaluation (M&E) architecture in place, which has evidently hindered the systematic documentation of progress and achievements. There is variation in the degree of successfulness and progress of activities under the different components. Planning and reporting documents did not contain disaggregated data by sex and vulnerable groups. Under the new agreement, there is room to include stand-alone gender and/or human rights criteria or indicators. Most interventions (except recruitment processes) were well and timely executed with no significant reported delays. The adopted actions and strategies were mostly relevant to achieving the expected results in the concerned areas during the implementation period. With a marked flexibility and ability to adapt to unexpected threats or risks – especially since the onset of the COVID pandemic - the team managed to achieve the desired results during the implementation period. Ideally, it would have been extremely useful to interview the individuals (National Project Managers) who managed the project’s interventions from the UNDP and the MEP. This step could have helped in gaining more comprehensive insights about the outcomes. However, this step was not possible since these people were not on board anymore. This is considered to have a significant impact on the consistency of the assessment of progress under the effectiveness criterion, which exclusively relied on the insights gathered from both secondary and primary research. As work mostly included elements of soft assistance (e.g., standards development, training, policy advocacy) it makes the application of conventional efficiency indicators to these areas not feasible (i.e. procurement of assets, comparative analysis to alternatives…).

Recommendations
1	UNDP Recommendation 1: Continue supporting the project in resource mobilization and design for the coming phase to reach sustainability of existing successful initiatives considering UNDP’s core role serving scalability and higher impact.
2	UNDP Recommendation 2: Maintain, develop, and further systemize standards and procedures (meetings, assessments/situational analysis/contingency plans, etc.) to ensure readiness to quickly identify and address unexpected/emerging issues with a solid documentation and clear formulation of the framework and corresponding tools.
3	UNDP Recommendation 3: Activate the project board/ steering committee and recruit a Chief Technical Advisor who can bring to the table the comparative and competitive advantage of UNDP (as part of the UN system) - as compared to private management consulting firms-, and to adopt an implementation approach that is based on a solid system-level analysis which will most likely translate into strategic results and policy-focused interventions.
4	UNDP Recommendation 4: Develop rigorous transparent communication processes and mechanisms with the national partner where all credible concerns related to the project’s implementation are investigated and addressed appropriately. In addition, develop an onboarding package for the new staff which includes the organizational mandatory trainings that are related to the rules and regulations and job responsibilities and ensure that the staff always conduct themselves in a professional and disciplined manner.
5	UNDP Recommendation 5: Focus future efforts towards improving the business maturity of existing successful initiatives rather than initiating new ones using a phased approach with clear outcome level indicators and exit strategies, unless otherwise is identified based on the continuous assessment of needs or as deemed necessary. Under the new agreement, there is an imminent need for improving business stability, active incentives, and strategies to engage beneficiaries.
6	UNDP Recommendation 6. Develop a measurement standard to systematically track the type, quality, and effectiveness of its contribution to gender results that also captures the context of change and the degree of its contribution to that change.
7	UNDP Recommendation 7. Establish a M&E system based on a shared understanding of goals and intended results, the architecture of the new agreement and the approved multi-year work plan, in addition to its corresponding quantitative and qualitative reporting tools, in view of monitoring and measuring achievements and progress of the project.
8	UNDP Recommendation 8: Support the modernization of knowledge and the transformation of conventional capacity development tools to uphold introduced changes (Prioritization of pilot initiatives through current/available proposals suggested by the MEP to embed knowledge at the institutional level. For example, creation of knowledge hub/repository, community of national economists, pool of experts for potential assignment, gradual production of research outputs).
9	MEP Recommendation 9: Continue enhancing and improving the business/investment environment.
10	MEP Recommendation 10: Assign a thematic division within the MEP as a focal point (keep recruitment and performance appraisal issues within the Directorate of the Human Resources Development Department).
11	MEP Recommendation 11: Support the development of liaison and coordination mechanisms and processes between the MEP and the UNDP, with clear and demarcated lines of reporting and decision-making capabilities across and between established structures, to jointly: 1) Regularly align expectations, 2) Analyze needs/emerging needs, 3) Strategize, prioritize, and plan integration into the broader MEP strategy, 4) Explore partnerships and synergies options, 5) Coordinate mobilization and allocation of resources, 6) Manage, monitor, and evaluate interventions, delivered capacity building interventions, and generated knowledge products, and 7) Agree on information sharing at both the institutional (internal) and large audience (external) levels.

Final evaluation Ministry of Economy and Planning Project

Share

Conclusion

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us