PAR project evaluation

Report Cover Image
Evaluation Plan:
2017-2022, Montenegro
Evaluation Type:
Final Project
Planned End Date:
12/2021
Completion Date:
12/2021
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
20,000

Share

Document Type Language Size Status Downloads
Download document OP-31 X 21 PAR Evaluation Report.pdf related-document English 723.66 KB Posted 450
Download document ToR Project Evaluation_PAR_June11.docx tor English 83.92 KB Posted 453
Download document OP-31 X 21 PAR Evaluation Report.pdf report English 723.66 KB Posted 463
Title PAR project evaluation
Atlas Project Number: 00110139
Evaluation Plan: 2017-2022, Montenegro
Evaluation Type: Final Project
Status: Completed
Completion Date: 12/2021
Planned End Date: 12/2021
Management Response: Yes
Focus Area:
  • 1. Governance
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.2.1 Capacities at national and sub-national levels strengthened to promote inclusive local economic development and deliver basic services including HIV and related services
  • 2. Output 2.2.1 Use of digital technologies and big data enabled for improved public services and other government functions
SDG Goal
  • Goal 16. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels
  • Goal 17. Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development
SDG Target
  • 16.6 Develop effective, accountable and transparent institutions at all levels
  • 17.18 By 2020, enhance capacity-building support to developing countries, including for least developed countries and small island developing States, to increase significantly the availability of high-quality, timely and reliable data disaggregated by income, gender, age, race, ethnicity, migratory status, disability, geographic location and other characteristics relevant in national contexts
Evaluation Budget(US $): 20,000
Source of Funding: Government
Evaluation Expenditure(US $): 14,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Olivera Puric International Evaluator SERBIA
GEF Evaluation: No
Key Stakeholders: Ministry of Public administration, EU, State institutions, municipalities
Countries: MONTENEGRO
Lessons
1.

Through synergetic layering of the groundwork, trust-building and piloting of different activities and approaches, the project was catalytic in furthering UNDP and public administration reform programming and in inviting strong donor interest

The project achievements also provided the basis for exploring higher-level policy options including the opportunities for improving the policy making process related to interconnectivity between the eGovernance and key areas of the public administration reform.


2.

The capacity development of national partners at the systemic, organisational and individual levels should be seen as an investment in both future programming and as a means of maximising partnerships. The capacity development should be seen as a long-term effort that needs to be embedded in broader change processes that are owned and driven by those involved, that are context-specific and that are as much about changing values and mindsets through incentives, as they are about acquiring new skills and knowledge. Also, UNDP should make a distinction between functional and technical capacities and support the development of both. The exact mixture of capacities to be addressed through a capacity development response will depend on the outcome of a capacity assessment. This will further contribute towards the long-term sustainability of the project results.


3.

The project requires standardised mechanisms for learning, in particular from its monitoring and evaluation efforts that can be reflected both in the project implementation, as well as fed into the CPD programmatic cycle. The next project cycle should factor in a midterm evaluation as a way to (re)adjust the course of action during the project implementation towards transformative results.


4.

The project should develop a systematic mechanism to capture lessons learned and preserve the project results to the extent possible. This includes preparation of analysis, case studies, lessons learned reports, document gathering etc.


5.

As a result of the analysis and assessments undertaken a more informed project can be designed, which is realistic and attainable given the operational realities, the opportunities, and constraints on the ground. A small set of high quality, measurable indicators should be developed and reviewed to ensure that the indicators are not only clearly defined but are also representative, reliable, and feasible. A greater use of qualitative indicators that measure perceptions and behaviours at the outcome level, as opposed to quantitative indicators that measure activities at the output level, will likely better capture project progress and results, as well as contributions towards the outcomes and impact.


Findings
1.

The project design reflects UNDP positioning

UNDP has a strong comparative advantage within the Public Administration sector and is recognized by the national partner as the only viable partner in the Public Administration Reform Sector, particularly in relation to eGovernance, human resource management including performance management, accountability and transparency of services. That position should be leveraged and capitalized on in the next programming cycle.

The understanding of the importance of the mainstreaming of the position of the UNDP as the key partner of Government of Montenegro in the public administration reform sector for the next programmatic cycle should inform the investment of CORE funds.

In addition to a clearly defined thematic focus UNDP should strategically use the existing momentum to elevate from a stand-alone project to a programmatic platform.


2.

Strategic Prioritisation

The evaluator finds that the project has a clear results framework, based on a coherent narrative and profile that is also linked to the CPD, and is attractive to national partners, donors, and other development partners.

As stated, the Project successfully mitigated the suboptimal implementation of the optimisation plan at the local level (output 1) and facilitated the interconnectivities between key state registers via SISEDE (output 2). The evaluator also finds the there is space for some programmatic revision and strategic re-focusing in terms of reformulating future outputs and the rationale behind the project, in order to ensure that the outputs are delivered and contribute to higher-level outcomes in a coherent manner, and that the project is successfully implemented.

Indicators should be revised to better capture the achievements of the project and their contribution to higher-level outcomes. There should be greater emphasis on qualitative indicators, which capture the voice of people, and in particular the most vulnerable, which would provide a clear pathway between activity – output – outcome – impact, as well as show progress towards change. In view of the fact that the project is constantly evolving to needs and context, it is recommended that the project set tailor made outcome level indicators, that would provide the goalposts for the project, while the output level indicators could be more flexible and adapted to circumstances as they change. It is recommended to revise the project efficiently so that the Project Board could approve the revisions, and in the most cost-effective and efficient manner.


3.

Communication of Results

The evaluator finds that communication with the national partners and donor took place on a regular, even daily basis. Crucially, the regular and open communication did allow for sufficient flexibility in terms of adapting the project activities in a substantial manner during the project implementation period. This allowed UNDP to align the project with the changing local political, legal and institutional context and design additional activities that were more suited and more achievable given the realities of the operational context.

The evaluator finds that communication mechanisms are very strong, however, communication could be further strengthened, through improving dialogue and the exchange of information, to ensure the quality assurance of the project results. UNDP needs to be continuously involved in dialogue with the project partners proactively addressing the challenges and limitations. A regular and active exchange with partners, with transparent communication of both positive and negative experiences will lead to sustainable partnership relations.

The evaluator finds that UNDP has successfully integrated the project into its wider portfolio, however it should go beyond just the communication of results and develop a strategic communication plan to raise the partnership profile.


4.

Project Expertise

National partners highly valued the project staff’s capacity, expertise, experience as well as commitment. The technical knowledge and skills of the project staff are assessed as excellent and fit for the task at hand. The project staff is also recognized as the ones that encourages innovative thinking and provide space for creative input and that value input from stakeholders and partners across the sector.

In addition to the support provided in implementation of the project activities the national partners find that the project staff were always ready to assist above and beyond in the development of policies, leadership skills, partnering abilities and institutional absorption capabilities in order to achieve and sustain development results.

At the same time the evaluator finds that the external experts did not always provide the needed technical support at the right level. As per the national partners’ view the external experts did not always have sufficient local contextual knowledge. Consequently, this created situation where some of the analytical findings and recommendations were not fully implementable.


5.

Project Approach

The evaluator recognises that all project partners are satisfied with the flexibility and responsiveness of UNDP to their needs. This has positioned UNDP as a preferred partner. However, the downside of this approach is that the assistance provided can be perceived as fragmented and developed on an ad hoc basis. It is perceived among some partners that UNDP has not fully maximised strategic cooperation and partnerships. This can potentially dilute UNDP’s positioning as a policy partner since it prevents the adequate strategic profiling of the organisation, meaning that UNDP is sometimes perceived as an organiser of events or provider of a logistical support while it is striving to be recognised as a leader, or master in public administration reform in Montenegro.

Further, the changing normative, institutional, and political context within which the project was implemented required implementing partners to keep funding modalities that allowed for a more responsive, flexible and adaptive approach to programmatic activities.

While the project has already taken steps towards developing a more partner-orientated focus, in the next phase the project should move this a step beyond. This should include building a higher degree of trust, adequately communicating and managing expectations and limitations, seizing opportunities and creating solutions.


6.

Sustainability

The evaluator finds that the project has very strong sustainability potential though the project duration was originally projected for 24 months (May 2018 - May 2020) and then extended until October 2021.

The ensured long-term relevance related to UNDP support to the Government of Montenegro in developing the new PAR strategy 2022 - 2026, generated systemic institutional buy-in and operationalisation of the EU Sector Budget Support financing instrument present strong evidence of the achieved sustainability.

However, the evaluator finds that although the project partners have been long-standing UNDP partners, partnership modalities were not maximally explored. The commitment of partners, including their financial commitment, is necessary to achieve greater sustainability of the project results and full national ownership of the project.

From the very first stages of the project design process, consideration should be given to the project’s exit strategy, and to the sustainability aspects of each of the project’s activities. These should be detailed in the project document, but regularly reviewed throughout the implementation period, to capture changing realities and ensure that maximum national ownership is achieved. The exit strategy should specify the transition arrangements to sustain and/or scale-up results, as relevant. It should describe how national capacities would be strengthened and monitored as relevant.


7.

The 2030 Agenda

While the project is highly relevant and tailored to the national context, including the EU Accession process and in particular political criteria related to the public administration reform, there is potential to position the project more strategically within the national and global sustainable development context. Although the EU accession process provides the key reform agenda for Montenegro, there are additional international commitments, which Montenegro is obligated to fulfil. The evaluator finds that the project could have been more closely aligned to the 2030 Agenda and the Sustainable Development Goals, which provides UNDP’s thematic niche for strategic positioning as a policy partner.

Montenegro has pledged itself internationally to meet the goals and targets of the 2030 Agenda, and any future programming should be positioned in such a way to assist Montenegro in fulfilling these international commitments. Particular reference should be made to SDG 16 and in particular targets 16.6 - Develop effective, accountable and transparent institutions at all levels; and 16.7 - Ensure responsive, inclusive, participatory and representative decision-making at all levels; and their corresponding indicators.


Recommendations
1

It is recommended to explore linkages and synergies with other projects so that this project can be elevated as a programmatic platform. The evaluator recommends that in future programming, UNDP improve the strategic linkages between the project components and partners through designing it in an innovative manner as a “platform project“, where UNDP is gathering strategic partners around the same goal, without being at the centre of the project design. This project design is particularly applicable in complex thematic areas, such as public administration reform, which has multiple stakeholders and multiple development challenges. This offers on the one hand the possibility for donors to access and cover more partners within the same framework and on the other hand facilitates horizontal cooperation among national partners. However, the complexity of partnerships requires both individual tailor-made approaches towards certain project components but also a strategic overview of the entire intervention. There is a risk that although the projects are within the same framework they operate as silos.

2

It is recommended that the project review its strategic direction allowing for some prioritisation of activities, while retaining its ability to be flexible and opportunistic. In order to provide some strategic direction, the evaluator recommends that the future programming use eGovernance as an enabler for programming in the area of public administration reform.

However, using eGovernment as the enabler requires going beyond the IT systems, tools and Infrastructure, because this does not automatically result in the streamlining and re-organization of government functions and transformation of roles and responsibilities. This needs to be part of a larger transformation process that underpin the human resource management, transparency and accountability.

At the same time during the next programmatic cycle a more holistic approach, that aims to balance a focus on internal reform with a concern for the changing relations between public administrations and citizens and business, will be needed. Functional assessments, as a tool of structural reform, need to provide the analytical basis and guide for programming and implementation. That includes identifying: a) redundant functions; b) duplication between and within institutions at central and local levels; c) missing functions as well as the review of the match between the function and the staff engaged in performing.

3

The communication between the project and partners should reflect work processes and achieve complementarity and increase programmatic synergies. The evaluator also recommends that project staff take a strategic, solution-orientated approach, which will further position the project and move from the communication of activity results towards communication of the strategic impact of results.

4

In the future special attention should be given to the process of selecting external consultants. The external consultants should be carefully selected with appropriate local and regional expertise and experience so they could provide expertise calibrated to the local context.

5

UNDP needs to enhance its strategic approach in future programming in addressing partners needs and institutional priorities though adaptive programming, based on the experiences gained during the implementation of the project. This highlighted the need for responsive, flexible, and adaptive implementation based on the changing context and as such, UNDP should consider a move towards even more adaptive programming. This will allow for the shifting of priorities and resources where results are not being achieved. This approach will require strong and measurable system-based indicators and rigorous monitoring, including a regularly updated risk management framework to make adjustments to programming on a regular basis, but will allow for the potential of better results with the same resources. These should be systematically programmed and budgeted to allow for review, reflection and adaptation as required.

The next phase of the project should be looking more at incubating and seeding in the public administration reform sector. This means the project is investing resources in multiple, concurrent small level pilots that may be based on successful approaches from within Montenegro or from external sources with similar contexts, and then to see which are able to be adapted to work in this country and in this political system.

It is also recommended that during incubating and seeding UNDP further explore the cooperation with civil society organisations that could be instrumental in testing certain solutions and providing real time data that could be feed in into the project implementation.

6

In the next programmatic phase, it is strongly recommended that the issues of sustainability and national ownership are looked at in a more comprehensive way from the planning stage. UNDP should strive to synchronise capacity development process with the Government strategic and planning processes. It may be feasible to incorporate the capacity development process at the level of programmes/projects, across the formulation, implementation and review. Approaching capacity development through this process lens provides a rigorous and systematic way of supporting it; improves the consistency, coherence and impact of efforts; and helps promote a common frame of reference for a programmatic response to capacity development and the sustainability prospects.

7

It is recommended to use the 2030 Agenda as a cohesive element in exploring and maximizing organic synergies between the project components and partners. This would also contribute to raising the profile of future project activities and results at the national level and assist Montenegro in meeting its international obligations, as well as strengthening UNDP’s strategic position as the key partner.

1. Recommendation:

It is recommended to explore linkages and synergies with other projects so that this project can be elevated as a programmatic platform. The evaluator recommends that in future programming, UNDP improve the strategic linkages between the project components and partners through designing it in an innovative manner as a “platform project“, where UNDP is gathering strategic partners around the same goal, without being at the centre of the project design. This project design is particularly applicable in complex thematic areas, such as public administration reform, which has multiple stakeholders and multiple development challenges. This offers on the one hand the possibility for donors to access and cover more partners within the same framework and on the other hand facilitates horizontal cooperation among national partners. However, the complexity of partnerships requires both individual tailor-made approaches towards certain project components but also a strategic overview of the entire intervention. There is a risk that although the projects are within the same framework they operate as silos.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued through the new UN Sustainable Development Cooperation Framework (UNSDCF) and CPD 2022-2026 formulation process. In the new CPD programme cycle (2022-2026), UNDP will focus on designing Public Administration Reform as a ‘’platform project’’.  By mainstreaming of the position of the UNDP as the key partner of Government of Montenegro in the public administration reform sector, UNDP will use the existing momentum to elevate from a stand-alone project to a programmatic platform.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
New CPD/programming developed and validated with national partners
[Added: 2021/12/15] [Last Updated: 2022/10/13]
RR/Team Leaders 2022/10 Completed In both new UNSDCF and CPD (2023-2027), UNDP has designed Public Administration Reform as a ‘’platform project’’. The CPD formulation was explicit: 24. UNDP will support public administration reform to design services that are more people-centred, gender responsive, adaptable and efficient. The primary focus will be on improving inclusive local and national strategic planning, participatory results-based management, and better monitoring and evaluation, with data and evidence at the centre of decision-making on policies. UNDP will support the government in pursuing financial technology solutions to improve the transparency and efficiency of public finance management. The work will be closely coordinated with initiatives implemented by the Organisation for Economic Co-operation and Development through its initiative supporting improved governance and management and the Regional School for Public Administration. History
2. Recommendation:

It is recommended that the project review its strategic direction allowing for some prioritisation of activities, while retaining its ability to be flexible and opportunistic. In order to provide some strategic direction, the evaluator recommends that the future programming use eGovernance as an enabler for programming in the area of public administration reform.

However, using eGovernment as the enabler requires going beyond the IT systems, tools and Infrastructure, because this does not automatically result in the streamlining and re-organization of government functions and transformation of roles and responsibilities. This needs to be part of a larger transformation process that underpin the human resource management, transparency and accountability.

At the same time during the next programmatic cycle a more holistic approach, that aims to balance a focus on internal reform with a concern for the changing relations between public administrations and citizens and business, will be needed. Functional assessments, as a tool of structural reform, need to provide the analytical basis and guide for programming and implementation. That includes identifying: a) redundant functions; b) duplication between and within institutions at central and local levels; c) missing functions as well as the review of the match between the function and the staff engaged in performing.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued through the new programming cycle. The strategic re-focusing will be made in terms of reformulating future outputs and the rationale behind the project, in order to ensure that the outputs are delivered and contribute to higher-level outcomes in a coherent manner, and that the project is successfully implemented.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Both recommendations will be addressed during next CPD programme cycle
[Added: 2021/12/15]
Responsible unit Team Leader and Programme Management 2026/12 Initiated History
3. Recommendation:

The communication between the project and partners should reflect work processes and achieve complementarity and increase programmatic synergies. The evaluator also recommends that project staff take a strategic, solution-orientated approach, which will further position the project and move from the communication of activity results towards communication of the strategic impact of results.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued through the next programming cycle. Communication will be further strengthened, through improving dialogue and the exchange of information, to ensure the quality assurance of the project results. UNDP will be continuously involved in dialogue with the project partners proactively addressing the challenges and limitations. A regular and active exchange with partners, with transparent communication of both positive and negative experiences will lead to sustainable partnership relations. UNDP will go beyond just the communication of results and develop a strategic communication plan to raise the partnership profile.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Develop a strategic communication plan to raise the partnership profile and strengthening communication beyond communicating results
[Added: 2021/12/15]
Responsible unit Team Leader and Programme Management 2026/12 Initiated
4. Recommendation:

In the future special attention should be given to the process of selecting external consultants. The external consultants should be carefully selected with appropriate local and regional expertise and experience so they could provide expertise calibrated to the local context.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued immediately.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Ensure that external experts always provide the needed technical support at the right level
[Added: 2021/12/15]
Responsible unit Programme Management 2027/12 Initiated
5. Recommendation:

UNDP needs to enhance its strategic approach in future programming in addressing partners needs and institutional priorities though adaptive programming, based on the experiences gained during the implementation of the project. This highlighted the need for responsive, flexible, and adaptive implementation based on the changing context and as such, UNDP should consider a move towards even more adaptive programming. This will allow for the shifting of priorities and resources where results are not being achieved. This approach will require strong and measurable system-based indicators and rigorous monitoring, including a regularly updated risk management framework to make adjustments to programming on a regular basis, but will allow for the potential of better results with the same resources. These should be systematically programmed and budgeted to allow for review, reflection and adaptation as required.

The next phase of the project should be looking more at incubating and seeding in the public administration reform sector. This means the project is investing resources in multiple, concurrent small level pilots that may be based on successful approaches from within Montenegro or from external sources with similar contexts, and then to see which are able to be adapted to work in this country and in this political system.

It is also recommended that during incubating and seeding UNDP further explore the cooperation with civil society organisations that could be instrumental in testing certain solutions and providing real time data that could be feed in into the project implementation.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued through the new programming cycle.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Building a higher degree of trust, adequately communicating and managing expectations and limitations, seizing opportunities and creating solutions.
[Added: 2021/12/15]
Responsible unit Team Leader and Programme Management 2023/12 Initiated
6. Recommendation:

In the next programmatic phase, it is strongly recommended that the issues of sustainability and national ownership are looked at in a more comprehensive way from the planning stage. UNDP should strive to synchronise capacity development process with the Government strategic and planning processes. It may be feasible to incorporate the capacity development process at the level of programmes/projects, across the formulation, implementation and review. Approaching capacity development through this process lens provides a rigorous and systematic way of supporting it; improves the consistency, coherence and impact of efforts; and helps promote a common frame of reference for a programmatic response to capacity development and the sustainability prospects.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued through the new programming cycle. The exit strategy will specify the transition arrangements and sustain and/or scale-up results, as relevant. It will describe how national capacities would be strengthened and monitored as relevant.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Develop the exit strategy
[Added: 2021/12/15]
Responsible unit Team Leader and Programme Management 2023/12 Initiated
7. Recommendation:

It is recommended to use the 2030 Agenda as a cohesive element in exploring and maximizing organic synergies between the project components and partners. This would also contribute to raising the profile of future project activities and results at the national level and assist Montenegro in meeting its international obligations, as well as strengthening UNDP’s strategic position as the key partner.

Management Response: [Added: 2021/12/15]

The recommendation is accepted and will be pursued through the new programming cycle. Particular reference will be made to SDG 16 and in particular targets 16.6 - Develop effective, accountable and transparent institutions at all levels; and 16.7 - Ensure responsive, inclusive, participatory and representative decision-making at all levels; and their corresponding indicators.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Align future programming more closely to the Agenda 2030
[Added: 2021/12/15]
Responsible unit Team Leader and Programme Management 2023/12 Initiated

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org