Review of the UNDP Evaluation Policy

Report Cover Image
Evaluation Plan:
2014-2017, Independent Evaluation Office
Evaluation Type:
Others
Planned End Date:
10/2014
Completion Date:
10/2014
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
300,000

Share

Document Type Language Size Status Downloads
Download document Baasterl-UNDP-Evaluation_PR_2014.pdf report English 2045.90 KB Posted 172
Title Review of the UNDP Evaluation Policy
Atlas Project Number:
Evaluation Plan: 2014-2017, Independent Evaluation Office
Evaluation Type: Others
Status: Completed
Completion Date: 10/2014
Planned End Date: 10/2014
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.1. National and sub-national systems and institutions enabled to achieve structural transformation of productive capacities that are sustainable and employment - and livelihoods- intensive
  • 2. Output 2.1. Parliaments, constitution making bodies and electoral institutions enabled to perform core functions for improved accountability, participation and representation, including for peaceful transitions
Evaluation Budget(US $): 300,000
Source of Funding:
Evaluation Expenditure(US $): 300,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: No
Key Stakeholders:
Countries: GLOBAL
Lessons
Findings
Recommendations
1

The evaluation policy should require management to introduce and enforce effective quality assurance systems for decentralized evaluations, with verification by the Independent Evaluation Office (IEO) and penalties on units that do not comply with standards. This should be supported by updated and additional guidelines and an assessment by the Executive Board of the value added to available management information by large numbers of low budget evaluations of variable quality.

2

In the absence of timely feedback from management parties, IEO should be given clear authority to proceed according to the pre-established timetable with all steps in the independent evaluation process. Enduring objections by management to evaluation findings or recommendations should be reserved for inclusion in the management response and should not be allowed to delay national workshops or report publication. UNDP units (such as country offices) that hinder the progress or completion of independent evaluations should be penalized through a “red flag” system.

 

 

3

The Executive Board should amend the evaluation policy to specify the lead role of the Board in recruitment procedures for the Director of IEO, the duration of the post (subject to performance assessment), renewal processes and duration, and power of the Director to report directly to the Board as necessary. These measures would strengthen the structural independence of the Office, in keeping with its new title.

4

The review notes that the evaluation units of the associated funds and programmes have developed in quite different ways and recommends that the policy should incorporate new and flexible approaches as follows: evaluation units should be required either to submit their independent evaluations to IEO for quality assurance (to ensure comparable “best international evaluation standards”); or to collaborate directly with IEO to manage and report on their independent evaluations. Associated with these measures, it is also recommended that the policy should require IEO to pay more systematic attention to the contribution of the associated funds and programmes to UNDP results in all of its independent evaluations.

5

The section of the policy on definitions should be replaced by a more general text indicating that IEO will periodically update and disseminate current evaluation topics and definitions on the basis of best international standards, through operational handbooks and other appropriate means.

IEO response: IEO agrees that including certain definitions within the evaluation policy may not allow sufficient flexibility to adapt to changing contexts. Nonetheless, certain other concepts should be clearly defined in the evaluation policy to ensure a common understanding across the entire organization. These include the concept of independent evaluation as well as distinctions between evaluation and the areas of monitoring and audit.

All definitions, either within the policy or elsewhere (for example, in guidance) should be consistent across UNDP (especially with those used in results-based management), and should reflect international norms and standards, including those set by the United Nations Evaluation Group.

1. Recommendation:

The evaluation policy should require management to introduce and enforce effective quality assurance systems for decentralized evaluations, with verification by the Independent Evaluation Office (IEO) and penalties on units that do not comply with standards. This should be supported by updated and additional guidelines and an assessment by the Executive Board of the value added to available management information by large numbers of low budget evaluations of variable quality.

Management Response: [Added: 2017/01/24] [Last Updated: 2019/02/06]

UNDP recognizes that despite consistent improvements to the evaluation policy over the past several years, there are weaknesses to be addressed in the current system for decentralized evaluations. UNDP welcomes the recommendation to further strengthen the provisions for effective quality assurance for decentralized evaluations. UNDP is committed to supporting further substantial improvements in the decentralized evaluation function. Challenges to be addressed include the evaluability of programmes, the independence of evaluations, the credibility and utility of decentralized evaluations, and the system for quality assurance. UNDP will address these challenges using a three-pronged approach:

(a) The Evaluation Policy will be revised to safeguard independence and impartiality so as to improve the credibility and hence the utility of decentralized evaluations. The new policy will clarify roles and responsibilities at all levels and introduce checks and balances. In revising the policy the alternative approaches suggested by the review, and other options, will be carefully considered.

(b) UNDP will ensure that regulations and procedures are in place to protect evaluators from undue interference and promote an evaluation culture that nurtures transparency, acceptance of criticism and willingness to learn from evaluation. UNDP recognizes that what external consultants consider ‘challenges’, such as inadequate time and resources, are normal challenges facing all evaluations. UNDP will work to enforce a zero-tolerance policy towards any undue interference in the work of independent evaluation consultants. Among the tools being considered are: systematic monitoring of the evaluation process by regional evaluation specialists; a written record of all stakeholders’ comments on draft reports (much of which is verbal today); assurance that any delay in payment is fully justified; assurance that reports are not ‘edited’ by UNDP; and the establishment of a ‘hotline’ where inappropriate behaviour and actions can be reported. The latter two are already being implemented pursuant to instructions from the Administrator

(c) UNDP will continue and strengthen its initiatives to invest in institutional and human capacities so as to further improve the quality of evaluations. UNDP management and IEO will together identify what should be done to enhance the capacity of UNDP staff to design, commission, and manage evaluations and ensure that contracted consultants have the ability to conduct credible evaluations of high quality. Monitoring will be designed to provide reliable information for day-to-day management, performance assessment, and evaluations. Training of evaluation specialists and managers will be intensified, and a certification and accreditation system will be instituted for evaluation specialists. External evaluation consultants will be drawn from a roster kept and quality-assured by IEO. UNDP will continue to train staff in results-based management, using the approaches and commitments made in the new Handbook on Planning, Monitoring and Evaluating for Development Results, which will be updated periodically to take account of evolving concepts and methods. Systems, tools and practices will be further revised, with clear accountability mechanisms built in to hold senior managers in country offices, regional centres and headquarters units accountable for improvements and results focus across the whole programming cycle. Special emphasis will be placed on systematically integrating the gender-related norms and standards of the United Nations Evaluation Group (UNEG) into decentralized evaluations in order to meet the requirements of the United Nations system-wide action plan by 2017.

(d) In addition to the above measures, UNDP will consider whether producing 300 decentralized evaluations per year, many with small budgets, is appropriate. One option might be to replace several of these with ‘project completion reports’ or ‘end-of-project reviews’, and conduct a limited number of strategically important evaluations.

IEO response: IEO agrees with the policy review team that management needs to introduce effective quality assurance systems for decentralized evaluation. Such a system needs to be complemented with (a) an effective oversight system that assess the quality and impartiality of decentralized evaluations; and (b) a system of incentives to strengthen the evaluation culture across the organization.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Prepare inputs to the joint drafting process of the new evaluation policy aimed at safeguarding and strengthening the independence and impartiality of decentralized evaluations, including specific provisions for effective quality assurance systems.
[Added: 2017/02/05]
UNDP Bureau for Policy and Programme support (BPPS) UNDP regional bureaus In consultation with IEO 2016/09 Completed The new Evaluation Policy, which stipulates clear evaluation principles under section III, was adopted in September 2016.
1.2 Develop and integrate new provisions on simplified recruitment of evaluation consultants from an IEO roster to support decentralized evaluations ensuring that evaluation consultants have competencies to apply the UNEG gender standards, among other requirements.
[Added: 2017/02/05] [Last Updated: 2018/04/04]
UNDP Bureau of Management (BMS)/Office of Human Resources (OHR) UNDP/BPPS In consultation with IEO 2018/12 Completed IEO is to expand the evaluation consultant roster and update, including establishing more rigorous procedures for vetting roster candidates and rating past performance by Dec 2017. UNDP will then further develop and integrate the provisions for simplified recruitment processes. IEO has in 2017 revised the consultancy database, and linked the evaluators with the evaluation QA ratings provided. Discussions are ongoing regarding payment of evaluators and other related functions. History
1.3 Develop and adopt provisions to mandate the establishment of an advisory group (of IEO staff and UNDP country- regional- and headquarters-level evaluation specialists) to provide quality assurance and ensure compliance of decentralized evaluations.
[Added: 2017/02/05] [Last Updated: 2018/04/04]
UNDP/BPPS UNDP regional bureaus In collaboration with IEO 2019/12 Completed The Evaluation Policy from 2016 clearly outlines the roles and responsibilities of various parties in UNDP/the IEO. The IEO is responsible for quality assessing evaluations, while UNDP is responsible for quality assurance. UNDP (RBx and COs under the guidance and overview of BPPS) collaborates closely with the IEO in reviewing and monitoring the quality and compliance of decentralized evaluations. UNDP has also created a community of practice/ knowledge platform in order to support management of the range of decentralized evaluation activities, and help standardizing approaches to evaluations, including monitoring of follow-up to management responses, enhancing access to, and management of lessons learned, and in particular the lessons learned from evaluations. History
1.4 Continuously develop and update results-based management-related training programmes; update the online course for evaluation practitioners. The advisory group will integrate gender expertise.
[Added: 2017/02/05] [Last Updated: 2018/03/21]
UNDP/BPPS UNDP/BMS In collaboration with IEO 2018/03 Completed UNDP and the IEO jointly conducted RBM/ M&E trainings for all five regions in 2016 and 2017. Discussions are ongoing regarding how to provide strategic and effective capacity building trainings going forward. Discussion is ongoing regarding including M&E training into a Project Management Certification. History
A new system for the oversight of decentralized evaluation established. The system would cover compliance with the evaluation policy, assessment of the quality and impartiality of decentralized evaluations, assessment of the quality of evaluation plans, and implementation of the strategy to strengthen quality assurance of decentralized evaluations.
[Added: 2019/02/06] [Last Updated: 2019/02/07]
IEO 2017/06 Completed Enhancement of evaluation quality assessment system was completed in 2015. IEO presented the Review of the Quality Assessment Of 2016 Decentralized Evaluations to the Executive Board in January 2018. The report can be accessed at: http://web.undp.org/evaluation/documents/quality-assessment/DE_QA_2016.pdf History
Strategy to effectively communicate revised evaluation policy to all staff and partners developed and implemented.
[Added: 2019/02/07]
BPPS and IEO 2015/12 Completed IEO actions: revised Evaluation Policy announced on the IEO web site, posted to the ERC , and explained in a series of regional workshops with CO M&E focal points in 2017 covering all 5 region.
2. Recommendation:

In the absence of timely feedback from management parties, IEO should be given clear authority to proceed according to the pre-established timetable with all steps in the independent evaluation process. Enduring objections by management to evaluation findings or recommendations should be reserved for inclusion in the management response and should not be allowed to delay national workshops or report publication. UNDP units (such as country offices) that hinder the progress or completion of independent evaluations should be penalized through a “red flag” system.

 

 

Management Response: [Added: 2017/01/24] [Last Updated: 2019/02/06]

UNDP management acknowledges that there have been instances of delay in finalizing IEO evaluation reports. The revised policy will specify time limits for the finalization of reports and management responses. The establishment of timelines for commenting on drafts and preparing management responses will prevent UNDP units from delaying the finalization of reports.

IEO response: IEO recognizes that undertaking an independent evaluation confers responsibilities on all parties concerned. Those responsibilities are set out in the evaluation policy. It also appreciates that UNDP management has intervened to reduce delays in the evaluation process. IEO has made efforts to ensure that more effective processes that do not lead to delays. IEO will continue to strengthen and codify its procedures to facilitate the rapid identification of solutions where disputes occur.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Prepare inputs to the new evaluation policy, spelling out precise timelines for the finalization of reports and related management responses.
[Added: 2017/02/05] [Last Updated: 2017/02/06]
UNDP BPPS UNDP regional bureaus In consultation with IEO 2016/09 Completed The new Evaluation Policy, which contains guidance on evaluation procedures and quality assurance, was adopted by the Executive Board in September 2016. History
New process guidelines for assessments of development results incorporate actions to reduce the likelihood of disputes and address them effectively should they occur.
[Added: 2019/02/07]
IEO 2014/11 Completed IEO postponed this effort pending revision of the ADR process and expansion to full coverage of all CPDs prior to renewal. The new (ICPE) format has been launched in 2018, including new process guidelines. The full coverage format and procedures for ICPEs, together with a new Evaluation Charter completed and launched in 2018 are expected to address and minimise such disputes.
New thematic evaluation process guidelines incorporate actions to reduce the likelihood of disputes and address them effectively should they occur.
[Added: 2019/02/07]
IEO 2019/04 Overdue-Initiated IEO prioritized the ICPE revisions and guidelines, then decentralised evaluation guideline production. The next step is a revision of thematic guidelines and procedures ongoing and scheduled for completion in April 2019. In the intervening period since the new policy was approved their have been no significant disputes on thematic process.
3. Recommendation:

The Executive Board should amend the evaluation policy to specify the lead role of the Board in recruitment procedures for the Director of IEO, the duration of the post (subject to performance assessment), renewal processes and duration, and power of the Director to report directly to the Board as necessary. These measures would strengthen the structural independence of the Office, in keeping with its new title.

Management Response: [Added: 2017/01/24] [Last Updated: 2019/02/06]

The present UNDP evaluation policy states that “[t]he Executive Board of UNDP/UNFPA is the custodian of the evaluation policy” and stipulates that the Executive Board: (a) approves the evaluation policy and considers the annual reports on its implementation; (b) ensures the independence of the evaluation function by (i) approving annually the costed programme of work for the Evaluation Office and (ii) reviewing and advising on the appointment, renewal and dismissal of the Director of the Evaluation Office”. While these provisions already provide for the lead role of the Board in recruitment procedures for the IEO Director, UNDP senior management is committed to reviewing these provisions as part of the revision of the UNDP evaluation policy.

IEO response: IEO acknowledges that UNDP management has undertaken a number of measures in recent years to strengthen the structural and operational independence of IEO and agrees with the recommended approach of codifying those measures in the revised evaluation policy. IEO agrees with the additional amendments to the policy recommended by the review team, including the lead role to be played by the Executive Board in the recruitment, renewal and dismissal of the IEO Director.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Prepare inputs to the new evaluation policy that further clarify the lead role of the Executive Board in recruitment, performance assessment, the duration and renewal of the post, and dismissal processes for the Director of IEO, as well as direct reporting of the IEO Director to the Board.
[Added: 2017/02/05] [Last Updated: 2017/02/06]
UNDP/BPPS, in consultation with IEO 2016/09 Completed The new Evaluation Policy, which contains clear guidelines and roles and responsibilities of the various stakeholders, was adopted by the Executive Board in September 2016. History
Amend policy to implement the recommendation
[Added: 2019/02/07]
Evaluation policy drafting team 2014/12 Completed The new Evaluation Policy was adopted by the Executive Board in September 2016. http://web.undp.org/evaluation/policy.shtml
4. Recommendation:

The review notes that the evaluation units of the associated funds and programmes have developed in quite different ways and recommends that the policy should incorporate new and flexible approaches as follows: evaluation units should be required either to submit their independent evaluations to IEO for quality assurance (to ensure comparable “best international evaluation standards”); or to collaborate directly with IEO to manage and report on their independent evaluations. Associated with these measures, it is also recommended that the policy should require IEO to pay more systematic attention to the contribution of the associated funds and programmes to UNDP results in all of its independent evaluations.

Management Response: [Added: 2017/01/24] [Last Updated: 2019/02/06]

UNDP management and UNCDF welcome the proposals to formalize a quality assurance for UNCDF evaluations. UNDP management and UNV welcome the proposal to establish a mechanism for direct collaboration with IEO to manage and report on UNV independent evaluations. UNV further welcomes the proposal to systematically review UNV contributions to UNDP results in all of its independent evaluations. 

IEO response: IEO can incorporate additional work but will require commensurate resources and authority. IEO can include UNV and UNCDF evaluations in the quality assessment systems and, as with UNDP decentralized evaluations, work together with UNCDF and UNV to introduce processes to ensure impartiality. IEO can also conduct a small number of strategic independent evaluations, if requested by the Executive Board, to include in its medium-term evaluation plan. UNCDF will still need its own decentralized evaluation quality assurance capacity in order to continue its work in this area, especially in embedding evaluation in its projects and programmes.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Prepare inputs to the new evaluation policy further clarifying the roles and responsibilities of UNCDF senior managers for evaluation.
[Added: 2017/02/05] [Last Updated: 2017/02/06]
UNCDF Evaluation Unit UNV Results Management Support Section 2016/09 Completed The new Evaluation Policy, including guidelines for UNCDF management, was adopted by the Executive Board in September 2016. History
4.2 Prepare inputs to the new evaluation policy establishing new mandatory criteria for UNCDF and UNV evaluations.
[Added: 2017/02/05]
UNCDF Evaluation Unit UNV Results Management Support Section 2016/09 Completed The new Evaluation Policy, which contains criteria for UNCDF and UNV evaluations, was adopted by the Executive Board in September 2016.
4.3 Prepare inputs to the new evaluation policy establishing mandatory quality assurance mechanisms for UNCDF evaluations.
[Added: 2017/02/05] [Last Updated: 2017/02/06]
UNCDF Evaluation Unit 2016/09 Completed The new Evaluation Policy was adopted by the Executive Board in September 2016. Section IV. in the Policy states the mandatory QA mechanisms. History
4.4 Prepare inputs to the new evaluation policy establishing working arrangements for IEO to manage and report on UNV independent evaluations.
[Added: 2017/02/05] [Last Updated: 2017/02/06]
UNV Results Management Support Section 2016/09 Completed The new Evaluation Policy, which contains articles on working arrangements between IEO and UNV, was adopted by the Executive Board in September 2016. History
Include UNCDF and UNV in reinvigorated decentralized evaluation quality assessment system
[Added: 2019/02/06]
IEO 2015/01 Completed IEO has revamped its quality assessment system to include UNV and UNCDF evaluation. All evaluation completed by UNV and UNCDF since 2015 are now quality assessed by IEO. Quality Assessment reports are available in ERC.
Undertake key strategic independent evaluations, if requested by the Executive Board, to include in a revised medium-term programme
[Added: 2019/02/07]
IEO 2017/12 Completed The IEO has carried out strategic evaluations under its multi-year work plan, 2018-2021, approved by the Board in response to the new Strategic Plan. Work in 2018 included two additional evaluations on pooled finance and operational services, which were undertaken in response to Board interest, and in light of the UNDS reform effort. History
Amend assessment of development results guidance to explicitly examine the contribution of partnerships with UNCDF and UNV
[Added: 2019/02/07]
IEO 2014/11 Completed New ICPE guidance and procedures developed in 2018 make reference to UNCDF and UNV.
5. Recommendation:

The section of the policy on definitions should be replaced by a more general text indicating that IEO will periodically update and disseminate current evaluation topics and definitions on the basis of best international standards, through operational handbooks and other appropriate means.

IEO response: IEO agrees that including certain definitions within the evaluation policy may not allow sufficient flexibility to adapt to changing contexts. Nonetheless, certain other concepts should be clearly defined in the evaluation policy to ensure a common understanding across the entire organization. These include the concept of independent evaluation as well as distinctions between evaluation and the areas of monitoring and audit.

All definitions, either within the policy or elsewhere (for example, in guidance) should be consistent across UNDP (especially with those used in results-based management), and should reflect international norms and standards, including those set by the United Nations Evaluation Group.

Management Response: [Added: 2017/01/24] [Last Updated: 2019/02/06]

UNDP welcomes the recommendation to revise the section on definitions. UNDP recognizes the need to update the section and will ensure that the new evaluation policy is in line with UNEG definitions, norms and standards.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Prepare inputs into joint drafting process of the new evaluation policy (provisions related to definitions, norms and standards).
[Added: 2017/02/05]
UNDP/BPPS 2016/09 Completed The new Evaluation Policy was adopted by the Executive Board in September 2016.
Revised policy to include definitions of only selected key terms. Other definitions to be provided in guidance.
[Added: 2019/02/07]
Evaluation Policy drafting team (policy), and IEO (guidance) 2015/12 Completed The new Evaluation Policy was adopted by the Executive Board in September 2016.The new policy reduces the amount of detail as suggested, with further guidance then developed through the Evaluation Charter and through revised UNDP evaluation guidelines in January 2019.
All guidance to use a consistent set of terms
[Added: 2019/02/07]
IEO, BPPS 2018/12 Completed Revised evaluation guidelines, in line with the Evaluation Policy were published in January 2019, with regional CO M&E focal point workshops scheduled in 2019 to review and elaborate. http://web.undp.org/evaluation/guideline/

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org