Independent Review of the UNDP Evaluation Policy

Report Cover Image
Evaluation Plan:
2009-2013, Independent Evaluation Office
Evaluation Type:
Others
Planned End Date:
01/2010
Completion Date:
01/2010
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
180,000

Share

Document Type Language Size Status Downloads
Download document Review_of_UNDP_Evaluation_Policy.pdf report English 1100.02 KB Posted 617
Title Independent Review of the UNDP Evaluation Policy
Atlas Project Number:
Evaluation Plan: 2009-2013, Independent Evaluation Office
Evaluation Type: Others
Status: Completed
Completion Date: 01/2010
Planned End Date: 01/2010
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
Evaluation Budget(US $): 180,000
Source of Funding:
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Team Leader
GEF Evaluation: No
Key Stakeholders:
Comments: This review was done by a team independent from the Evaluation Office.
Lessons
Findings
Recommendations
1 Evaluation recommendation 1. UNDP senior management must decide whether decentralized evaluation is of a high enough priority that it is willing to commit the focus and resources needed to implement the approaches envisaged in the new Handbook on Planning, Monitoring and Evaluating for Development Results. Management should do this by: (a) Acknowledging the magnitude of the challenge; (b) Taking a clear lead in ensuring that changes envisaged are implemented as quickly and effectively as possible; (c) Revising the UNDP Programme and Operations Policies and Procedures (POPP) where necessary, to ensure alignment between the handbook and policies and procedures; and (d) Defining the means, capacities and timeline required to implement the changes needed to strengthen the decentralized evaluation system, and ensure resources are allocated, implementation is properly monitored and corrective action taken, if needed. This recommendation will require changes in systems and practices across the whole planning and project cycle, with ?evaluation? being integrated into all new initiatives as they are being developed, as well as into staff appraisal systems.
2 Evaluation recommendation 2. The senior management of UNDP will need to build on the opportunities to build national leadership and ownership in evaluation. In responding to changes introduced by the UNDG on results reporting and results frameworks used at country level, the senior management of UNDP will need to revise the new Handbook for Planning, Monitoring and Evaluating for Development Results, the UNDP Programme and Operations Policies and Procedures, and other tools and guidelines. These revisions should also recognize an ongoing need for the Evaluation Office to draw upon this data for the assessment of development results and corporate level evaluations, which are still required to meet corporate level accountability and learning objectives. The Evaluation Office should reassess its methodological guidance in the light of these changes, and work within UNEG to craft a common response on how to balance corporate and national-level needs for evaluative evidence.
3 Evaluation recommendation 3. The Executive Board should amend the evaluation policy to institutionalize the independence of the Evaluation Office. This would include: (a) Recruitment of the Director of the Evaluation Office. In the current policy, the Administrator appoints the Director of the Evaluation Office, in consultation with the Executive Board, and ensures there is no conflict of interest in employment, including limiting the term of appointment to four years, renewable once, and barring re-entry into the organization. Institutionalization of independence would be significantly strengthened if the role of the Executive Board in appointing the Director were strengthened and clearly spelled out in the policy; (b) Recruitment of the Evaluation Office Staff. As long as standard UNDP human resources practice is followed, the power of the QUARRY to overrule decisions made by the Director should be removed; (c) Clarifying relationships. The relationship of the Director of the Evaluation Office to other senior managers within UNDP, and on what basis the Director would participate in strategic planning processes within UNDP, should be clarified; (d) Expanding career opportunities for the Evaluation Office Staff. The possibilities for Evaluation Office staff to be mainstreamed into core positions in the wider organization, with opportunities to rotate and be promoted in line with standard UNDP procedures, should be strengthened; and (e) Budget. The process for setting the budget of the Evaluation Office is currently described in broad terms within the present policy, whereby the Administrator is responsible for provision of sufficient resources, and the budget is negotiated biannually with the Bureau of Management. The guiding principle should be that the budget is set to adequately fund the work programme agreed upon between the Evaluation Office and the Executive Board. Good practice would be for the budget to be approved by the Executive Board as part of the Evaluation Office workplan approval process.
4 Evaluation recommendation 4. The Evaluation Office to consider the degree to which the present approach to development and implementation of assessment of development results truly contributes to country ownership. Particular issues that should be considered are: participation of government partners in deciding the scope and focus of the assessment of development results; and consideration of the recommendations of, and management response to, the evaluation.
5 Evaluation recommendation 5. The Evaluation Office should work through UNEG to clarify (a) the comparative advantage of UNDP in building capacity for evaluation at the country level; and (b) what steps should be taken by the Evaluation Office and the respective country programmes to build on this comparative advantage.
6 Evaluation recommendation 6. The Executive Board should consider requesting a review to be presented to the Board in 2012 covering: ? The degree to which the roles and responsibilities laid out in the 2007 POPP and 2009 Handbook have been fully and effectively implemented; ? The degree to which adoption of approaches advocated in the Handbook has strengthened (i) RBM and (ii) decentralized evaluation at the country level; ? The degree to which independence of the EO has been institutionalized; ? The degree to which the policy has been implemented and made a positive contribution in UNDP?s associated funds and programmes; ? Whether an effective approach to strengthening country ownership and capacity building has been identified and is being implemented.
1. Recommendation: Evaluation recommendation 1. UNDP senior management must decide whether decentralized evaluation is of a high enough priority that it is willing to commit the focus and resources needed to implement the approaches envisaged in the new Handbook on Planning, Monitoring and Evaluating for Development Results. Management should do this by: (a) Acknowledging the magnitude of the challenge; (b) Taking a clear lead in ensuring that changes envisaged are implemented as quickly and effectively as possible; (c) Revising the UNDP Programme and Operations Policies and Procedures (POPP) where necessary, to ensure alignment between the handbook and policies and procedures; and (d) Defining the means, capacities and timeline required to implement the changes needed to strengthen the decentralized evaluation system, and ensure resources are allocated, implementation is properly monitored and corrective action taken, if needed. This recommendation will require changes in systems and practices across the whole planning and project cycle, with ?evaluation? being integrated into all new initiatives as they are being developed, as well as into staff appraisal systems.
Management Response: [Added: 2011/03/29] [Last Updated: 2011/03/30]

UNDP senior management is fully committed to improving the decentralized evaluation function. Challenges to be addressed include the evaluability of programmes, minimum requirements for evaluation coverage, conduct of outcome evaluations, evaluation funding and follow-up. UNDP will continue to train staff on RBM, based on the approaches and commitments made in the new Handbook on Planning, Monitoring and Evaluating for Development Results. Systems, tools and practices will be further revised, with clear accountability arrangements built in to hold senior managers in country offices, regional centres and headquarters units accountable for improvements and results focus across the whole programming cycle. From 2010, the Management Group will discuss management responses to all independent evaluations. ADRs and their management response will be reviewed in the context of senior management discussions of results and progress against the Strategic Plan. Incentive and performance systems will be revised to motivate demand for evaluative evidence and encourage sharing and follow-up of lessons. The Evaluation Office will continue to support UNDP senior management by providing regular briefings to the Management Group and Operations Group on recurring findings and recommendations from independent evaluations. It will continue to set standards and prepare guidelines for independent and decentralized evaluations. The Evaluation Office is developing an e-learning course. Additionally, it will continue to collaborate with the United Nations Evaluation Group (UNEG) on United Nations?wide training in evaluation. The Evaluation Office will also continue to maintain the online, publicly accessible database ? the Evaluation Resource Centre (ERC) ? which makes available all evaluation plans, reports, terms of references, management responses and a tracking system for follow-up actions. This aids knowledge sharing and learning, information management and oversight. The Evaluation Office will continue to support the evaluation community of practice through ERC and continued hosting of EvalNet, the e-network on evaluation, which currently has 1,311.members.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Make full implementation of the evaluation plan attached to the country, regional and global programme documents the basis for evaluation compliance, and present it to the Board fully costed
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Regional Bureaux 2011/02 Completed It is mandatory for UNDP to prepare and present a costed evaluation plan to its Executive Board as an annex to the global, regional, country and other programme documents that are submitted for approval. History
1.2 Revise guidance to include more options for decentralized evaluation coverage, including notably greater use of midterm outcome-oriented programme reviews. Outcome evaluation requirements will be revised to favour participation in multi-partner evaluations as agreed with national partners, while ensuring independence and credibility of the process
[Added: 2011/03/30] [Last Updated: 2012/02/01]
Regional Bureaux 2010/01 Completed
1.3 Revise guidance on minimum requirements for evaluation coverage at the decentralized level
[Added: 2011/03/30] [Last Updated: 2012/02/01]
Operations Support Group, Evaluation Office 2010/12 Completed
1.4 Present options for funding of decentralized evaluations to ensure full funding of CPD evaluation plans and flexibility, depending on country need
[Added: 2011/03/30] [Last Updated: 2011/04/11]
Operations Support Group, Evaluation Office 2010/01 Completed
1.5 Revise the POPP and other tools and guidelines where necessary to ensure alignment between the Handbook, United Nations Development Group (UNDG) guidelines, good practice and policies and procedures
[Added: 2011/03/30] [Last Updated: 2012/02/01]
Operations Support Group, Bureau for Development Policy/Capacity Development Group, Evaluation Office 2010/09 Completed
1.6 Maintain regional rosters of pre-qualified evaluation experts and institutes
[Added: 2011/03/30] [Last Updated: 2013/03/21]
Regional Bureaux, with Evaluation Office quality control 2011/12 No Longer Applicable The Evaluation Office has established a new online evaluation expert roster with the regional bureaux and regional service centres, establishing a common platform that will consolidate various evaluation expert rosters and other parallel systems maintained by different programme units.
1.7 Develop an online course on evaluation
[Added: 2011/03/30] [Last Updated: 2013/03/21]
Evaluation Office 2010/10 Completed
1.8 Manage ERC, vetted Roster and EvalNet and set evaluation standards and guidelines
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Evaluation Office 2011/03 Completed History
1.9 Amend existing tools such as the balanced score card to create incentives for achieving quality standards in RBM, including monitoring and evaluation
[Added: 2011/03/30] [Last Updated: 2013/03/21]
Bureau of Management with Operations Support Group and Evaluation Office 2011/12 Completed New Balanced Scorecard indicators on the quality of results reporting and on the quality of decentralized evaluations have been introduced.
1.10 Recruit at least one (where appropriate two) monitoring and evaluation advisors in each region
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Regional Bureaux 2011/03 Completed History
1.11 Strengthen RBM training and institutionalize follow-up
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Regional Bureaux 2011/03 Completed The Evaluation Policy provides clear guidance regarding roles and responsibilities of various parties in UNDP. RBM trainings are undertaken on a regular basis. History
1.12 Ensure the systematic use of monitoring and evaluation knowledge for programme adjustments during the programme cycle, development of new programmes and advisory services and knowledge products
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Regional Bureaux, Bureau for Development Policy, Bureau for Crisis Prevention and Recovery, Partnerships Bureau 2011/03 Completed History
2. Recommendation: Evaluation recommendation 2. The senior management of UNDP will need to build on the opportunities to build national leadership and ownership in evaluation. In responding to changes introduced by the UNDG on results reporting and results frameworks used at country level, the senior management of UNDP will need to revise the new Handbook for Planning, Monitoring and Evaluating for Development Results, the UNDP Programme and Operations Policies and Procedures, and other tools and guidelines. These revisions should also recognize an ongoing need for the Evaluation Office to draw upon this data for the assessment of development results and corporate level evaluations, which are still required to meet corporate level accountability and learning objectives. The Evaluation Office should reassess its methodological guidance in the light of these changes, and work within UNEG to craft a common response on how to balance corporate and national-level needs for evaluative evidence.
Management Response: [Added: 2011/03/29] [Last Updated: 2011/03/30]

UNDP agrees with this recommendation. As part of oversight of country offices, Regional Bureaux will ensure increased broad-based involvement of national stakeholders and partners in the planning, management, conduct and use of evaluation, as advocated in the new Handbook for Planning, Monitoring and Evaluating for Development Results. Building on the new approaches to assessments of development results introduced by the Evaluation Office, guidance will be reinforced to encourage country offices to make more use of national systems (where appropriate), and country-led evaluations will be advocated as the option of choice wherever possible. In this regard, UNDP proposes to amend the evaluation policy to prioritize country-led approaches in the next generation of evaluations, as exemplified by the Government-led joint evaluation of the United Nations? role in South Africa and the country-led evaluations of Delivering as One pilots, which draw on the UNEG Framework Terms of Reference. UNDP will revise the Handbook, POPP and other tools and guidelines (where necessary) to ensure alignment with the UNDG guidelines. The revisions will tackle the current lack of methodological clarity to allow credible assessment of the UNDP contribution. For its part, the Evaluation Office will expand the evaluation methodology manual for all types of decentralized evaluations for programme units, including the outcome evaluation guidelines, in response to changes and trends at the UNDG level, particularly those stipulated in the new United Nations Development Assistance Framework guidelines. The Evaluation Office has revised approaches and methodology for independent evaluations (ADRs and thematic) accordingly. The revised ADR method manual was completed in April 2010.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Articulate steps to strengthen the engagement of government partners in the conduct of decentralized evaluations and management responses
[Added: 2011/03/30] [Last Updated: 2011/04/11]
Operations Support Group, Evaluation Office 2010/12 Completed
2.2 Expand methodology manual to address all types of decentralized evaluations
[Added: 2011/03/30] [Last Updated: 2012/02/01]
Evaluation Office 2010/10 Completed
2.3 Work with UNEG on country-level evaluation approaches
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Evaluation Office 2011/03 Completed History
3. Recommendation: Evaluation recommendation 3. The Executive Board should amend the evaluation policy to institutionalize the independence of the Evaluation Office. This would include: (a) Recruitment of the Director of the Evaluation Office. In the current policy, the Administrator appoints the Director of the Evaluation Office, in consultation with the Executive Board, and ensures there is no conflict of interest in employment, including limiting the term of appointment to four years, renewable once, and barring re-entry into the organization. Institutionalization of independence would be significantly strengthened if the role of the Executive Board in appointing the Director were strengthened and clearly spelled out in the policy; (b) Recruitment of the Evaluation Office Staff. As long as standard UNDP human resources practice is followed, the power of the QUARRY to overrule decisions made by the Director should be removed; (c) Clarifying relationships. The relationship of the Director of the Evaluation Office to other senior managers within UNDP, and on what basis the Director would participate in strategic planning processes within UNDP, should be clarified; (d) Expanding career opportunities for the Evaluation Office Staff. The possibilities for Evaluation Office staff to be mainstreamed into core positions in the wider organization, with opportunities to rotate and be promoted in line with standard UNDP procedures, should be strengthened; and (e) Budget. The process for setting the budget of the Evaluation Office is currently described in broad terms within the present policy, whereby the Administrator is responsible for provision of sufficient resources, and the budget is negotiated biannually with the Bureau of Management. The guiding principle should be that the budget is set to adequately fund the work programme agreed upon between the Evaluation Office and the Executive Board. Good practice would be for the budget to be approved by the Executive Board as part of the Evaluation Office workplan approval process.
Management Response: [Added: 2011/03/29] [Last Updated: 2011/03/30]

The following revisions to the policy are proposed: Evaluation Office Director: The Administrator will consult with the Executive Board on the appointment, renewal or dismissal of the Director. The term of the Director is four years, renewable only once. After the completion of his or her term, he or she cannot seek another appointment in the organization in order to avoid conflict of interest. The job description will be in line with the established UNEG Competencies for Heads of Evaluation, and the post will be advertised internally and externally. The normal UNDP recruitment procedures will be followed. The Administrator will perform the annual performance appraisal of the Director in the standard UNDP format, and the result will be shared with the Bureau of the Executive Board. There will be no review by the Career Review Group. Evaluation Office staff: Consistent with current practice regarding support staff, the Evaluation Office will continue to follow normal UNDP recruitment procedures. As now, professional staff job descriptions will be in line with the established UNEG Competencies for evaluators, and posts will be advertised internally and externally. The Evaluation Office will conduct technical written tests and technical interviews with external UNEG members. The panel for final, competency-based interviews will include other UNDP colleagues, including a representative from the Office of Human Resources. Throughout the process, the Office of Human Resources will guide the Evaluation Office to ensure fulfilment of UNDP policies and regulations. The Director of the Evaluation Office will take the final decision on selection after the Compliance Review Board has ensured that all UNDP policies and regulations have been followed, including those concerning gender and regional balance, and has validated the selection process. All Evaluation Office staff, including the Director, will be recruited under the terms of contract for UNDP and will adhere to the UNEG Ethical Guidelines and Code of Conduct for evaluators. The Evaluation Office staff will have the same obligations, rights and opportunities for career advancement as other UNDP staff. Relationship between the Evaluation Office Director and the rest of the organization and his/her role in decision-making: As now, the Director will not be part of the organization?s decision-making processes. Budget: The Executive Board will continue to approve the Evaluation Office work programme and approve its budget in the context of approval of the UNDP biennial support budget.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Amend the Evaluation Policy to specify that UNDP will consult with the Executive Board on the appointment, renewal or dismissal of the Director of the Evaluation Office
[Added: 2011/03/30]
Administrator 2011/03 Completed Consultation with the Executive Board will take place every four years when the term of the Director comes to an end, or as appropriate
3.2 Select Evaluation Office staff following UNDP procedures
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Evaluation Office Director 2011/06 Completed Per UNDP HR procedures and 2011, and 2016 revisions to the UNDP evaluation policy History
3.3 The Evaluation Office Director will not take part in the organization's decision-making processes
[Added: 2011/03/30] [Last Updated: 2018/04/04]
IEO 2011/03 Completed clarified in 2011 and 2016 revisions to the Evaluation Policy History
3.4 Continue to seek approval of the Evaluation Office budget by the Executive Board in the context of its approval of the UNDP biennial support budget
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Evaluation Office 2013/01 Completed During the relevant session of the Executive Board annual costed programme of work provided to the Board at its 1st regular session each year History
4. Recommendation: Evaluation recommendation 4. The Evaluation Office to consider the degree to which the present approach to development and implementation of assessment of development results truly contributes to country ownership. Particular issues that should be considered are: participation of government partners in deciding the scope and focus of the assessment of development results; and consideration of the recommendations of, and management response to, the evaluation.
Management Response: [Added: 2011/03/29] [Last Updated: 2011/03/30]

The Evaluation Office is introducing new approaches to assessment of development results. The Evaluation Office task manager conducts a preparatory mission to determine the most suitable approach for conducting the assessment in a given country. There are four broad approaches: (1) a team of independent consultant evaluators, led by an international team leader with the Evaluation Office task manager, conducts an evaluation; (2) joint evaluation with an independent national evaluation office and management by a joint evaluation management group (example: South Africa); (3) the government establishes a reference group to engage with the Evaluation Office in developing the terms of reference and drafting the report (examples: Ghana, Mongolia and Thailand); and (4) the Evaluation Office engages an independent national institution to carry out the evaluation (examples: China and Thailand). All assessments will continue to focus on UNDP accountability for its contribution to development results. In all cases, the independence of the evaluation team and the quality and utility of the evaluation will be guaranteed by the Evaluation Office. UNDP will build on these new approaches for decentralized evaluations to strengthen country ownership.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Pilot new approaches in the ADRs conducted during 2010-2011
[Added: 2011/03/30] [Last Updated: 2011/04/11]
Evaluation Office 2011/12 Completed
5. Recommendation: Evaluation recommendation 5. The Evaluation Office should work through UNEG to clarify (a) the comparative advantage of UNDP in building capacity for evaluation at the country level; and (b) what steps should be taken by the Evaluation Office and the respective country programmes to build on this comparative advantage.
Management Response: [Added: 2011/03/29] [Last Updated: 2011/03/30]

The UNDP comparative advantage is its ability to promote and coordinate South-South and trilateral cooperation in support of capacity building for evaluation at the country level. This will be facilitated by strengthening communities of practice in evaluation and maintaining regional rosters of evaluation experts and institutes in each region (see key action 1.6 above). Building on the excellent work already being done by some country offices, UNDP will strengthen efforts to support national evaluation capacity through South-South and trilateral cooperation, when appropriate. This will be determined country by county, based on needs and UNDP comparative advantage to provide such capacity development in the country. The Evaluation Office held a first conference on National Evaluation Capacities in Morocco in December 2009, which served as a forum for discussion of evaluation issues confronting countries and enabled participants to draw on recent and innovative experiences of other countries. The conference promoted an understanding of international standards in evaluation and advocated for evaluation to contribute to improvements in management for development results and in public accountability and learning. The conference prepared the ground for formulation of longer-term initiatives to strengthen national capacities for public policy evaluation through South-South (or trilateral) cooperation. A network has been established and annual conferences are foreseen.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Organize regular national evaluation capacity conferences
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Evaluation Office 2011/08 Completed national evaluation capacity conferences held every two years – rotating across each of the 5 regions. History
5.2 Support national evaluation capacity through South-South and trilateral cooperation
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Regional Bureaux, Country 2011/03 Completed This is currently being done across all regions History
5.3 Strengthen communities of practice in evaluation
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Regional Bureaux, regional service centres 2011/03 Completed Completed and ongoing History
6. Recommendation: Evaluation recommendation 6. The Executive Board should consider requesting a review to be presented to the Board in 2012 covering: ? The degree to which the roles and responsibilities laid out in the 2007 POPP and 2009 Handbook have been fully and effectively implemented; ? The degree to which adoption of approaches advocated in the Handbook has strengthened (i) RBM and (ii) decentralized evaluation at the country level; ? The degree to which independence of the EO has been institutionalized; ? The degree to which the policy has been implemented and made a positive contribution in UNDP?s associated funds and programmes; ? Whether an effective approach to strengthening country ownership and capacity building has been identified and is being implemented.
Management Response: [Added: 2011/03/29] [Last Updated: 2011/03/30]

UNDP will keep the Executive Board informed of progress in implementation of the key actions.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 Regularly inform the Executive Board of progress in the implementation of the key actions
[Added: 2011/03/30] [Last Updated: 2018/04/04]
Executive Office, Evaluation Office 2011/06 Completed annual reporting to the Board at each annual session through the Annual Reports on Evaluation History

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org