Review of the UNDP evaluation policy

Report Cover Image
Evaluation Plan:
2018-2021, Independent Evaluation Office
Evaluation Type:
Others
Planned End Date:
06/2019
Completion Date:
05/2019
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
100,000

Share

Document Type Language Size Status Downloads
Download document UNDP 2016 Evaluation Policy Review TOR.pdf tor English 158.42 KB Posted 31
Download document UNDP Evaluation Policy UNDP.pdf report English 726.12 KB Posted 60
Title Review of the UNDP evaluation policy
Atlas Project Number:
Evaluation Plan: 2018-2021, Independent Evaluation Office
Evaluation Type: Others
Status: Completed
Completion Date: 05/2019
Planned End Date: 06/2019
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Organisational Output 1.3 High quality audits and evaluations producing implementable solutions
Evaluation Budget(US $): 100,000
Source of Funding:
Evaluation Expenditure(US $): 95,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Osvaldo Feinstein Independent Consultant ofeinstein@yahoo.com
GEF Evaluation: No
Key Stakeholders:
Countries: GLOBAL
Lessons
Findings
Recommendations
1 An amendment to the 2016 UNDP evaluation policy should include a reference to the Charter of the Independent Evaluation Office and to the 2019 evaluation guidelines.
2 The principles of the evaluation policy should include an explicit reference to the 2030 Agenda for Sustainable Development, gender equality, diversity, inclusion, human rights and the private sector.
3 The planning process should involve consultation with stakeholders, and in all phases of the evaluation process it is important that evaluators engage with stakeholders and ensure not only that the national context is considered but also that the purpose, relevance and messages of the evaluations are communicated clearly and using a language that does not create unnecessary tensions.
4 The decision on what to evaluate should be made with an explicit statement of the purpose and potential use of the evaluations for strategic decision-making.
5

A technical reporting line of regional monitoring and evaluation (M&E) specialists to the IEO Director on evaluation issues would contribute to enhance the quality of decentralized evaluations. Additional funding from evaluation partnerships may be instrumental in developing arrangements to strengthen the oversight and support to decentralized evaluations and to make the Evaluation Resource Centre more useful, with a better use of quality assurance scores. The evaluation responsibilities of the regional M&E specialists should be enhanced.

6

The use of different and new types of evaluations and data-collection methods should be encouraged, including whenever appropriate a complexity and systems approach and paying attention to innovation and scaling-up.

7

After producing evaluations, efforts should be made to elaborate messages derived from the evaluations, including syntheses, showing trends/patterns based on granular data, that may be of interest to different audiences.

8

The requirement in the evaluation policy that a management response should be prepared for all evaluations and in a fixed period of time could be changed so as to alleviate the stress on management’s absorptive capacity. In the case of independent country programme evaluations (ICPEs), the new country programme could be considered as an option of a management response, whereas the Executive Board could allow extensions in the submission of management responses.

9

As country programme evaluations are no longer mandatory, whereas ICPEs are in the process of arriving at 100 per cent coverage, there could be some flexibility in the overall 1 per cent of allocation of funds for evaluation, by introducing a link between the 0.8 per cent for non-IEO evaluations and the evolution of the UNDP portfolio of activities and funds. This would be facilitated if UNDP were to introduce a budget line to accurately capture funds allocated to evaluation.  The ambivalence regarding the funding level should be eliminated by deleting the last part of the sentence in paragraph 26, “subject to availability”.

10

Given the structural problem with respect to independence posed by the existence of an AEAC that reports to the Administrator, potentially compromising the independence of the IEO Director, the AEAC should no longer be part of the UNDP evaluation architecture

11

An independent and external review of the evaluation function should be conducted every four years by an external team reporting to the Board.

Management Response Documents
1. Recommendation: An amendment to the 2016 UNDP evaluation policy should include a reference to the Charter of the Independent Evaluation Office and to the 2019 evaluation guidelines.
Management Response: [Added: 2019/05/29]

The proposal to include a reference to the Charter of the IEO and the revised evaluation guidelines to the Evaluation Policy is welcomed.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Prepare inputs to the revised evaluation policy
[Added: 2019/05/29] [Last Updated: 2019/07/26]
Bureau for Policy and Programme Support, IEO 2019/07 Completed The IEO have in close collaboration with BPPS, BMS and ExO revised the 2016 evaluation policy. The revised version of the policy has been submitted to the executive Board for consideration at the second regular session. History
2. Recommendation: The principles of the evaluation policy should include an explicit reference to the 2030 Agenda for Sustainable Development, gender equality, diversity, inclusion, human rights and the private sector.
Management Response: [Added: 2019/05/29]

Inclusion of a reference to the 2030 Agenda into the policy is supported. However, the policy already references sustainable development and the private sector (in section ‘Purpose of evaluation’, paragraph 6) and gender equality, equity and human rights (in section ‘Evaluation Principles’, paragraphs 7 and 10). UNDP and the IEO will carefully review the current language of the policy adding additional references only if necessary.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Prepare inputs to the revised evaluation policy
[Added: 2019/05/29] [Last Updated: 2019/08/15]
Bureau for Policy and Programme Support, IEO 2019/07 Completed The IEO have in close collaboration with BPPS, BMS and ExO revised the 2016 evaluation policy. The revised version of the policy has been submitted to the executive Board for consideration at the second regular session History
3. Recommendation: The planning process should involve consultation with stakeholders, and in all phases of the evaluation process it is important that evaluators engage with stakeholders and ensure not only that the national context is considered but also that the purpose, relevance and messages of the evaluations are communicated clearly and using a language that does not create unnecessary tensions.
Management Response: [Added: 2019/05/29]

This is a useful and welcomed proposal. Stakeholder engagement is emphasized in the recently revised UNDP evaluation guidance and is highlighted as well in United Nations Evaluation Group norms and standards.

Key Actions:

4. Recommendation: The decision on what to evaluate should be made with an explicit statement of the purpose and potential use of the evaluations for strategic decision-making.
Management Response: [Added: 2019/05/29]

UNDP and IEO agree that this is a useful proposal. As pertains to IEO corporate/thematic evaluations, such explicit statements are included in the brief notes on each planned thematic evaluation included in the multi-year evaluation plan which the IEO submits to the Executive Board following the launch of each new UNDP strategic plan. The revised evaluation guidelines require that all evaluation plans submitted to the Programme Appraisal Committee for review be accompanied by a brief text explaining the rationale for the evaluations in the plan (i.e., how they contribute to accountability, learning and the achievement of strategic results and how they will provide sufficient and balanced coverage of the programme unit’s areas of engagement).

Key Actions:

5. Recommendation:

A technical reporting line of regional monitoring and evaluation (M&E) specialists to the IEO Director on evaluation issues would contribute to enhance the quality of decentralized evaluations. Additional funding from evaluation partnerships may be instrumental in developing arrangements to strengthen the oversight and support to decentralized evaluations and to make the Evaluation Resource Centre more useful, with a better use of quality assurance scores. The evaluation responsibilities of the regional M&E specialists should be enhanced.

Management Response: [Added: 2019/05/29] [Last Updated: 2019/05/29]

UNDP and the IEO consider that creating a matrix management system for the current regional M&E focal points is untenable. The recommendation of having a direct reporting line of existing focal points will not work as these are not evaluation personnel, and dual reporting lines are problematic as they have multiple roles that would be in conflict with serving also as an independent evaluator.

Instead, in order to address the ongoing concerns about the decentralized evaluation system, UNDP and the IEO will consider extending IEO coverage and positions from headquarters to the regional hubs with the creation of a cadre of P4/P5 posts plus support staff. Regional IEO personnel will rotate across the system, meaning that expertise will be fully infused into the evaluation architecture.

To address the issues of quality and capacity, UNDP will continue to undertake capacity-building efforts together with the IEO, including through training and awareness-raising on the revised guidelines, in particularly with respect to the management process of evaluation; oversight and quality assurance responsibilities of regional M&E staff; and the accountability of senior management.

UNDP management will actively seek funding opportunities and new partnerships going forward. Resource mobilization is a prerequisite to allocate additional funding as requested in the recommendation, especially for small country offices which are already stretched in terms of human and financial resources. A resource mobilization strategy to guide such efforts will be developed.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Prepare inputs to the revised evaluation policy
[Added: 2019/05/29] [Last Updated: 2019/07/26]
Bureau for Policy and Programme Support, IEO 2019/07 Completed The IEO have in close collaboration with BPPS, BMS and ExO revised the 2016 evaluation policy. The revised version of the policy has been submitted to the executive Board for consideration at the second regular session. History
Organizational Performance Group to review evaluation performance annually to guide a programme of continuous improvement and enhance management/leadership awareness and accountability
[Added: 2019/05/29] [Last Updated: 2019/07/26]
Regional bureaux, Bureau for Policy and Programme Support 2021/12 Initiated History
Develop online courses on evaluation for managers, M&E staff and project staff (in IEO workplan for 2018-2019) and make them mandatory for appropriate categories of staff (UNDP).
[Added: 2019/05/29] [Last Updated: 2019/07/26]
IEO, Executive Office 2019/08 Overdue-Initiated History
Conduct training workshops and online webinars for M&E and other staff on the new evaluation guidelines
[Added: 2019/05/29] [Last Updated: 2019/05/31]
Bureau for Policy and Programme Support, Regional bureaux, IEO 2019/12 Initiated Regional workshop held in Istanbul in February 2019 for the RBEC region. Two webinars have been organized: 8 Feb: webinar on the evaluation guidelines. To be accessed here: https://youtu.be/hCOJwlIkloY 14 May: webinar on UNDP evaluation and gender quality and women's empowerment. To be accessed here: https://youtu.be/GhHD35JeH7E History
Develop a resource mobilization strategy for the evaluation function
[Added: 2019/05/29]
Bureau for Policy and Programme Support, Bureau of External Relations and Advocacy, IEO 2019/12 Not Initiated
6. Recommendation:

The use of different and new types of evaluations and data-collection methods should be encouraged, including whenever appropriate a complexity and systems approach and paying attention to innovation and scaling-up.

Management Response: [Added: 2019/05/31]

UNDP and the IEO welcome this recommendation. There have been significant changes made to UNDP programming approaches, and it is important that the evaluation approaches are able to recognize the different initiatives and approaches (such as for example portfolios, acceleration labs, etc.). The evaluation guidelines contain useful prescriptive guidance in this regard. UNDP welcomes the use of new methodologies by the IEO, and for evaluations to be forward-looking to inform future programming directions. UNDP highlights the importance of considering the country contexts, recognizing that there is a need to be flexible and use innovative evaluation approaches in crisis countries. 

Key Actions:

7. Recommendation:

After producing evaluations, efforts should be made to elaborate messages derived from the evaluations, including syntheses, showing trends/patterns based on granular data, that may be of interest to different audiences.

Management Response: [Added: 2019/05/31]

The recommendation is welcomed, and UNDP is committed to undertaking more analysis for organizational learning going forward.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
7.1. Annual analysis on evaluation-related issues – results achieved/lessons learned/ strengths and weaknesses – included as part of the management commentaries to the annual reports on evaluation
[Added: 2019/05/31]
Bureau for Policy and Programme Support 2020/07 Initiated
8. Recommendation:

The requirement in the evaluation policy that a management response should be prepared for all evaluations and in a fixed period of time could be changed so as to alleviate the stress on management’s absorptive capacity. In the case of independent country programme evaluations (ICPEs), the new country programme could be considered as an option of a management response, whereas the Executive Board could allow extensions in the submission of management responses.

Management Response: [Added: 2019/05/31] [Last Updated: 2019/05/31]

Management responses are an integral part of the evaluation process to contribute to programme/project implementation effectiveness and organizational accountability. UNDP maintains a high level of management response compliance at around 95 per cent or above for both independent and decentralized evaluations with a rolling implementation rate of agreed actions of around 70 per cent. UNDP will consider the capture of its response to ICPE recommendations through new country programme documents.  UNDP management reiterates its commitment that all independent thematic and decentralized evaluations have a management response.

Key Actions:

9. Recommendation:

As country programme evaluations are no longer mandatory, whereas ICPEs are in the process of arriving at 100 per cent coverage, there could be some flexibility in the overall 1 per cent of allocation of funds for evaluation, by introducing a link between the 0.8 per cent for non-IEO evaluations and the evolution of the UNDP portfolio of activities and funds. This would be facilitated if UNDP were to introduce a budget line to accurately capture funds allocated to evaluation.  The ambivalence regarding the funding level should be eliminated by deleting the last part of the sentence in paragraph 26, “subject to availability”.

Management Response: [Added: 2019/05/31] [Last Updated: 2019/05/31]

UNDP and the IEO do not agree with this recommendation, which is not clear and contains some inaccuracies. Programme units are required to submit a costed and timed evaluation plan to the Executive Board with each country, regional and global programme document considered for approval. Information regarding evaluation planning, budgeting and which evaluations are mandatory is noted in the UNDP evaluation guidelines, launched in January 2019. Consistent with the UNDP cost-recovery policy and procedures, allocations for decentralized evaluations are required to be included in programme and project budgets. The requirement to include evaluation within programme and project budgets is fully incorporated within the revised UNDP programme and project manual.

 

Per paragraph 26 of the evaluation policy, “at the organizational level, UNDP will aim at allocating 1 per cent of combined programmatic (core and non-core) resources to the evaluation function: with no less than 0.2 per cent reserved for the work of Independent Evaluation Office, subject to availability of resources”. The funding of IEO regional posts and activities noted in paragraph 11 of this response should be included in the IEO annual workplan as a specific segment and this segment would be funded from the relevant line in programmes and project budgets, to a maximum of 0.3 per cent (i.e., the combined total of 0.2 per cent from the UNDP institutional budget reserved for the work of the IEO and 0.1 per cent from programme and project budgets), and this maximum of 0.3 per cent would be within the envelope of 1 per cent combined programmatic (core and non-core) resources allocation for the evaluation function, as stipulated in the evaluation policy.

 UNDP will  investigate the feasibility of establishing a reporting mechanism to accurately capture evaluation expenditures, both human resources and evaluation costs.

There are no objections towards revising the evaluation policy to delete the last phrase “subject to availability”.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
9.1. Prepare inputs to the revised evaluation policy
[Added: 2019/05/31] [Last Updated: 2019/07/26]
Bureau for Policy and Programme Support, IEO 2019/07 Completed The IEO have in close collaboration with BPPS, BMS and ExO revised the 2016 evaluation policy. The revised version of the policy has been submitted to the executive Board for consideration at the second regular session. History
9.2. Investigate the feasibility of a reporting mechanism for evaluation expenditures
[Added: 2019/05/31]
Bureau for Management Services 2019/12 Initiated History
10. Recommendation:

Given the structural problem with respect to independence posed by the existence of an AEAC that reports to the Administrator, potentially compromising the independence of the IEO Director, the AEAC should no longer be part of the UNDP evaluation architecture

Management Response: [Added: 2019/05/31]

Both IEO and UNDP management recognize the importance of having an oversight body for evaluation and value the AEAC in terms of providing advisory and oversight support. The recommendation is therefore not accepted. However, it is recognized that the Committee is still evolving in its focus and will in the forthcoming period examine ways to strengthen the evaluation coverage within its work. 

Key Actions:

11. Recommendation:

An independent and external review of the evaluation function should be conducted every four years by an external team reporting to the Board.

Management Response: [Added: 2019/05/31]

UNDP and IEO concur with the recommendation and note that the evaluation policy has been reviewed twice since 2010 (this is the third review) while the evaluation function receives United Nations Evaluation Group peer reviews every four years. 

Key Actions:

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org