Joint Assessment of the Institutional Effectiveness of UNDP

Report Cover Image
Evaluation Plan:
2014-2017, Independent Evaluation Office
Evaluation Type:
Thematic
Planned End Date:
08/2017
Completion Date:
02/2017
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
150,000

Share

Document Type Language Size Status Downloads
Download document Joint_Institutional_Effective_Final_Report_2017.pdf report English 2899.38 KB Posted 771
Download document Terms of Reference.pdf tor English 169.27 KB Posted 120
Download document IA_Exec_Summary_2017.pdf related-document English 87.52 KB Posted 99
Title Joint Assessment of the Institutional Effectiveness of UNDP
Atlas Project Number:
Evaluation Plan: 2014-2017, Independent Evaluation Office
Evaluation Type: Thematic
Status: Completed
Completion Date: 02/2017
Planned End Date: 08/2017
Management Response: Yes
Corporate Outcomes (UNDP Strategic Plan 2008-20013)
Evaluation Budget(US $): 150,000
Source of Funding:
Joint Programme: No
Mandatory Evaluation: No
Joint Evaluation: Yes
  • Joint with UN Agencies
  • Joint with UNDP OAI
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: No
Key Stakeholders:
Lessons
Findings
Recommendations
1

Based on the content of the UNDAF and the guidance provided in the UNDP quality standards, country offices should ensure that the UNDP comparative advantage and value added are adequately identified by providing evidence as to why UNDP is better positioned than other institutions to implement a specific programme.

2

UNDP should ensure that all country offices fully understand and adequatelypractise the concept of theory of change during the programming process through a thorough assessment of the completeness and internal logic of the theory of change prior to submitting the CPDs to the Executive Board.

3

UNDP should assess the costs of implementing the new programme and project quality assurance system to determine whether and, if so, how the resource requirements of the reformed system can be sustainably met with costed plans for a phased implementation. Based on the budget available, UNDP should prioritize the quality elements to which country offices have to adhere fully.

4

UNDP should reassess the financial sustainability of the regional service centres/hubs model including the posting of the BPPS policy advisers.

5

UNDP should develop greater RBM expertise with improved focus on learning and knowledge management for enhanced effectiveness, shifting the focus from proving results to improving results. To effectively institutionalize RBM, capacity development needs to be delivered through a broad range of approaches and include all staff, from leadership and senior management to programme managers and associates. Capacity development should also extend to implementing partners, whose engagement is essential if national data sets are to improve and contribute to UNDP reporting requirements. Increased attention should also be given to promoting an organizational culture that uses more effectively the conclusions, recommendations and lessons learned from evaluations and audits to contribute to knowledge management and to feed strategic and timely decision-making.

6

UNDP leadership should prioritize investment in knowledge management, going beyond capturing best practices to using lessons learned from each context of success and failure to contribute to effectiveness and improve results. The role of leadership is pivotal in ensuring an enabling environment and support for UNDP to enhance engagement and communication to further develop a results-based culture throughout the organization that welcomes critical reflection on performance and effective knowledge management to improve results, where successes but also failures are important vehicles of learning. Leadership should effectively encourage a “results culture” which goes beyond reporting and understands RBM in terms of continuous organizational self-learning from both successes and failure and from innovation, and not just M&E for reporting purposes.

1. Recommendation:

Based on the content of the UNDAF and the guidance provided in the UNDP quality standards, country offices should ensure that the UNDP comparative advantage and value added are adequately identified by providing evidence as to why UNDP is better positioned than other institutions to implement a specific programme.

Management Response: [Added: 2017/02/14] [Last Updated: 2017/11/14]

UNDP appreciates the assessment’s acknowledgement of improvement in the quality of CPDs. This is the result of clearer quality standards, more effective country support and active engagement by senior management in the appraisal of new programmes. The programme appraisal process with both the preliminary project appraisal and headquarters Project Appraisal Committee has led to more strategic, relevant and better articulated programmes according to the feedback received from Member States and the recent RBM performance audit. The added value of the CPD is well reflected in the assessments of development results conducted by the IEO; all those covering the current Strategic Plan period concluded that the UNDP contribution to national results generally has been very strong.

 

The UNDAF is jointly formulated by the United Nations country team for integrated programme planning to align with national development plans and priorities. UNDP fully supports the UNDAF formulation and implementation processes in partnership with other United Nations agencies. The UNDAF outcomes are developed with the core criteria of the comparative advantage and value added of the United Nations system, including individual United Nations entities. This includes a reflection on mandate, technical expertise, proven record, volume and precedent, access to development finance, thought leadership and expressed preference by development partners including programme country counterparts. UNDP then derives its programme outcomes directly from this framework. There is evidence that the UNDAF outcomes to which UNDP contributes are firmly linked to the UNDP Strategic Plan and respond to the Sustainable Development Goals and other internationally agreed agendas. Every CPD submitted to the Executive Board in 2016 responded clearly to national priorities and the Sustainable Development Goals while being firmly embedded in the UNDP Strategic Plan, using evaluation and other evidence to help explain the UNDP comparative advantage and added value in the country. This included CPDs from all regions, including those for Eritrea, Lebanon, Montenegro, Suriname and Viet Nam.

 

UNDP agrees on the importance of clearly identifying its comparative advantage and value added in programme plans based on evidence, including in the UNDAF and CPD. The 2016  internal analysis of  lessons learned on programme quality concluded that while 82 per cent of new CPDs used evaluation to help explain the UNDP comparative advantage within the country's development sector, UNDP CPDs are still weak in identifying the organization's comparative advantage vis-à-vis other partners. In response, the guidance and template for CPDs was recently revised to include stronger guidance and clearer expectations on the added value, comparative advantage and partnership strategies of UNDP. The headquarters Project Appraisal Committee will look at this issue more closely starting in 2017. It is also worth noting that the CPDs are constrained by a 6,000-word limit, which has an effect on the issues included. UNDP would encourage that the word limit be increased in future, but agrees that the programme design must, despite the length limitation, deliberately consider comparative advantages that have been assured by the organization’s internal appraisal processes.

Guidance on developing an UNDAF has been prepared through an inter-agency process in which UNDP is an active member. The UNDAF guidance has been revised to better respond to the ambitions of the Sustainable Development Goals. UNDP helped to ensure that quality standards for the UNDAF will be introduced in this new guidance to address concerns relating to the weaknesses of UNDAF design. This includes highlighting the comparative advantages of the United Nations system and the added value of individual agencies. Theories of change are also being introduced for the UNDAF, which will help to ensure closer linkage between the UNDAF and agencies’ country programmes and include comparative advantage as a core criterion for selecting priority areas for the UNDAF.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Ensure programming guidance includes emphasis on how to adequately reflect the UNDP comparative advantage based on evidence.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2017/12 Initiated
1.2 The headquarters Project Appraisal Committee's quality assurance of draft country programmes to review the evidence provided in CPDs of why UNDP is better positioned than other institutions in the selected priority areas, as standard practice.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2017/09 Completed
1.3 Targeted roll-out of support to countries undergoing country programming processes to highlight how the theory of change methodology should be used to identify agency comparative advantage.
[Added: 2017/11/14]
Regional bureaux 2018/06 Initiated
2. Recommendation:

UNDP should ensure that all country offices fully understand and adequatelypractise the concept of theory of change during the programming process through a thorough assessment of the completeness and internal logic of the theory of change prior to submitting the CPDs to the Executive Board.

Management Response: [Added: 2017/02/14] [Last Updated: 2017/11/14]

UNDP developed new extended guidance on theory of change that was adopted corporately in December 2016. For the purpose of strengthening consistency in integrating programming, UNDP has led the joint United Nations Development Group effort to elaborate inter-agency guidance on developing theories of change for the UNDAF. Any innovation will take time to consolidate and UNDP remains committed to pursuing further innovation for development. UNDP will continue to integrate theory of Change into corporate training modules, which will be a key focus in 2017. In this regard, UNDP has made much progress in crafting higher-quality CPDs based on a clear development pathway. The headquarters Project Appraisal Committee is a rigorous process and reads CPDs for evidence of a theory of change among other dimensions of quality. This will continue to be strengthened in 2017, along with key corporate decision points before the headquarters Project Appraisal Committee.

 

As noted by the RBM performance audit, theory of change does not simply mean using the term "theory of change, but is rather about basing the CPD on a clear development logic using the theory of change methodology. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 The preliminary project appraisal of CPDs to include a review of the completeness and internal logic of the theory of change.
[Added: 2017/11/14]
Regional bureaux 2017/12 Initiated
2.2 Targeted roll-out of support to countries undergoing country programming processes to highlight the concept of theory of change and how to apply it when designing country programmes.
[Added: 2017/11/14]
Regional bureaux 2017/12 Initiated
3. Recommendation:

UNDP should assess the costs of implementing the new programme and project quality assurance system to determine whether and, if so, how the resource requirements of the reformed system can be sustainably met with costed plans for a phased implementation. Based on the budget available, UNDP should prioritize the quality elements to which country offices have to adhere fully.

Management Response: [Added: 2017/02/14] [Last Updated: 2017/11/14]

The new UNDP quality standards for programming are designed to facilitate learning and decision-making to improve quality over time, rather than functioning as a compliance-oriented checklist. The standards reflect the different attributes that contribute to quality programming, drawing on learning from various evaluations, audits, assessments and other reviews. The rating tool is designed to flag strengths and – more importantly – weaknesses, to help managers decide on what investments to make in order to improve quality. Project quality assurance is done in the corporate planning system, which means that the data can be analysed at the individual project level, across a country office programme portfolio, for projects in an entire region and for UNDP corporately. Overall performance is reviewed against the standards. If some attributes do not meet the minimum standards and cannot be readily addressed, then either a management plan is put in place to address the issues over time, or it is flagged as a programmatic risk to be monitored by management because it will affect the delivery of quality results. A management plan that reflects on actions that will be taken to improve quality based on the resources available is central to the assessment process. For example, in some crisis contexts there may not be sufficient time to prepare all of the analysis required to meet the UNDP standards for many attributes of quality programming. This does not mean that important early recovery work cannot begin. On the contrary, the quality assurance process will help managers to flag issues that should be addressed once conditions permit, lest they be forgotten during implementation.

 

During the pilot phase, an internal UNDP assessment showed that on average it took programme staff about one hour per project per year to complete the quality assurance assessment. The greater investment is the time and resources required to improve quality, which is up to the manger to determine based on available resources. The quality assurance system will help the organization to better understand its portfolio at various levels and perform analysis that can help decision-making. UNDP will continue to assess the effectiveness of the quality assurance processes going forward, and adjust them to be in line with the new strategic plan.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Policy review of the programming standards following the closure of the 2016 project quality assurance.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2017/12 Initiated
3.2 Adjustment of the parameters of quality programming – if needed - in parallel to the period of the new strategic plan, 2018-2021, and provision of related training to country offices.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2021/12 Initiated By December 2021 (in parallel with the period of new strategic plan for 2018-2021)
4. Recommendation:

UNDP should reassess the financial sustainability of the regional service centres/hubs model including the posting of the BPPS policy advisers.

Management Response: [Added: 2017/02/14] [Last Updated: 2017/11/14]

The consolidation of the regional hubs in 2014 integrated programme support functions previously provided by the regional bureaux, BPPS policy and advisory support, management services from BMS and resource mobilization and communications support from the Bureau of External Relations and Advocacy.  UNDP welcomes the findings that country offices are satisfied with the quality and timeliness of support provided by the hubs, and that the integrated support provided has contributed to enhancements in quality.  As noted in the assessment report, the regional location of these services has proven to be cost-effective. UNDP management notes that regional hubs are not discrete entities with a separate financial sustainability model. In line with the new planning and budgeting approach process, the funding envelope for BPPS will be considered relative to and in collaboration with other bureaux each year, and expectedly will need to be adjusted as necessary in line with the overall resources available to UNDP.  The staffing configuration of BPPS will continue to evolve in alignment with the changing business model of UNDP to assure delivery of policy and programme support services that effectively address the changing needs and demands at global, regional and country levels. In line with the cost-recovery policy approved by the Executive Board, it is further noted that UNDP has instituted measures to ensure that the costs of project advisory and other services are charged to projects.  

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Regular review of 2017 institutional budget implementation, and development of 2018 budget allocations.
[Added: 2017/11/14]
Bureau for Management Services/ regional bureaux/ Bureau for Policy and Programme Support 2017/12 Initiated
5. Recommendation:

UNDP should develop greater RBM expertise with improved focus on learning and knowledge management for enhanced effectiveness, shifting the focus from proving results to improving results. To effectively institutionalize RBM, capacity development needs to be delivered through a broad range of approaches and include all staff, from leadership and senior management to programme managers and associates. Capacity development should also extend to implementing partners, whose engagement is essential if national data sets are to improve and contribute to UNDP reporting requirements. Increased attention should also be given to promoting an organizational culture that uses more effectively the conclusions, recommendations and lessons learned from evaluations and audits to contribute to knowledge management and to feed strategic and timely decision-making.

Management Response: [Added: 2017/02/14] [Last Updated: 2017/11/14]

The recommendation is broadly in line with the UNDP assessment of the current state of knowledge and learning in the organization. Rather than merely focusing on accountability and reporting, critical reflection and learning from past experiences to improve future results are an essential part of RBM. This will require increased investment in knowledge management capacity at country, regional and headquarters levels, training on relevant tools and methodologies and a culture that embraces constant reflection, learning and knowledge-sharing as the mandate of every team and individual staff member. Being aware of this need, already in 2016 UNDP strongly emphasized complementing the long-standing emphasis on RBM training and tools with a broad range of knowledge management approaches and solutions that analyse past experiences from the perspective of learning, enable staff to reflect and exchange on experiences on an ongoing basis, and identify resources (both content and people) that can help staff to apply this knowledge for improved results. These approaches included: analysis of lessons learned from results-oriented annual reports and decentralized evaluations from 2014 and 2015; the relaunch of the UNDP thematic knowledge networks; the roll-out of an upgraded knowledge sharing infrastructure; and the relaunch of the UNDP pubic library of knowledge products along with new mechanisms to measure their quality, reach and impact. UNDP is also investing in the development of an improved mechanism to track staff subject matter expertise across the organization. Recognizing the importance of “shifting the focus from proving results to improving results”, UNDP wishes to highlight that the effective use of conclusions, recommendations and lessons learned should go beyond only formal evaluations to include the more systematic leveraging of existing business processes (such as the project management cycle), informal knowledge content and staff interactions for future learning. To this end, UNDP will design and add a tool for capturing lessons learned in the corporate project management space.

 

Improving capacities of national implementing partners for RBM has been included in the new UNDP quality standards for programming, and should be considered in all projects where resources are available. The standards also include considering evidence learned from evaluations, audits and other sources of knowledge in the design and implementation of programmes and projects, and for decision-making. UNDP agrees that quality standards alone are not enough; it is crucial that staff and partners be capacitated and adequate resources made available in order to meet these standards. UNDP will therefore continue monitoring how strengthened capacities lead to a better RBM culture for learning and changing going forward.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Develop coherent RBM training modules in line with the new strategic plan, integrated results and resources framework, policies and procedures and incorporate an RBM training component in the induction courses for resident coordinators and resident representatives, training for country directors and deputy country directors, training for Junior Professional Officers and regional training and workshops.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2018/03 Initiated
5.2 Develop a RBM/programme management certificate for staff.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2018/12 Not Initiated
5.3 Deliver RBM/programme management training to programme staff, including managers.
[Added: 2017/11/14]
Regional bureaux 2021/12 Initiated Ongoing with no due date
5.4 Develop a staff expertise mapping and search system (“People Search” talent map) to enable offices to locate subject matter experts, including monitoring and evaluation.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group, Bureau for Management Services, Office of Human Resources 2018/12 Not Initiated
6. Recommendation:

UNDP leadership should prioritize investment in knowledge management, going beyond capturing best practices to using lessons learned from each context of success and failure to contribute to effectiveness and improve results. The role of leadership is pivotal in ensuring an enabling environment and support for UNDP to enhance engagement and communication to further develop a results-based culture throughout the organization that welcomes critical reflection on performance and effective knowledge management to improve results, where successes but also failures are important vehicles of learning. Leadership should effectively encourage a “results culture” which goes beyond reporting and understands RBM in terms of continuous organizational self-learning from both successes and failure and from innovation, and not just M&E for reporting purposes.

Management Response: [Added: 2017/02/14] [Last Updated: 2017/11/14]

UNDP agrees with the assessment that the role of leadership is pivotal in ensuring an enabling environment for knowledge management to improve results, which needs to include systematic reflection and learning from both successes and failures. UNDP wishes to highlight some of the important steps taken in the above direction, to promote and foster a results-based culture and to complement the already noted mechanisms and tools to improve the quality of programmes and projects. In this regard, in 2016 UNDP relaunched its thematic knowledge networks following a hiatus and consolidated its information and communication technology-based systems in support of knowledge management and knowledge sharing under the Microsoft Office 365 environment. In 2017, this will be complemented with dedicated training and outreach efforts to enable staff to use these mechanisms effectively for learning and knowledge exchange. Further, UNDP has taken practical steps to operationalize self-learning from experiments, from what works and what does not, through the establishment of the Innovation Facility in 2014. A key component of the mandate of this Facility is the provision of risk capital and advisory services to country offices to test new approaches to solve development problems. The Innovation Facility documents successes and lessons, based on failures, in its annual reviews and through frequent blogging. UNDP remains committed to testing innovations for development and to identify the drivers of failure for improved performance. The Facility's work has shown that a focus on failure is less conducive to source valuable lessons from country offices compared to qualitative questions that strive to identify actionable lessons.

 

In addition, UNDP introduced the Integrated Results and Resources Framework (IRRF) and the "Report Card" in the annual report of the Administrator and midterm review of the Strategic Plan, as the organization’s accountability mechanism against the Strategic Plan, 2014-2017.  The IRRF and Report Card, which serve as a base for performance dialogue with the Executive Board, pay special attention to "failure" by investigating root causes of low performance. The system will allow UNDP to improve the policy and oversight support from the headquarters and programming at country level. In the new strategic plan, UNDP will introduce a robust results framework and tighter linkages between results and resources, which will make all levels of the organization accountable for results and enhance the capacity to analyse the root causes of low performance. UNDP also continues to strengthen integrated reporting and work closely with country offices on strengthening the different aspects of reporting, and the understanding of how they complement each other. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 Production of the annual report of the Administrator, including the analysis of factors of low performance, to enable organizational learning and dialogue with Executive Board members to take concrete actions to achieve higher results.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2018/06 Initiated Annually by June
6.2 Strengthen the mechanism to analyse low performance and hindering factors in the monitoring and reporting systems.
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2018/06 Initiated
6.3 Develop and introduce the lessons learned "capture tab" into the corporate project management space (in the Corporate Strategic Planning System)
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group, Executive Office 2017/12 Completed
6.4 Curate growth of the UNDP knowledge networks as a vessel for sharing knowledge, and lessons learned and for critical self-reflection on programme effectiveness
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group and Professions 2017/12 Initiated
6.5 Produce a yearly lesson learned analysis of trends emerging from: (a) results-oriented annual report data; and (b) CPDs being submitted to the Executive Board
[Added: 2017/11/14]
Bureau for Policy and Programme Support/Development Impact Group 2017/12 Initiated

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org